LinformerEncoder Layer: No Linearisation?

[JBKnights]

Hello,

Sorry if this is a silly question, but looking at your code in ptr_base.py line 90 the LinformerEncoder layer doesn't seem to be implementing linear attention at all; what it seems to be doing instead is just performing regular multi-head attention. Is this the case, and if not where for the LinformerEncoder layers does the linearisation take place?

Thanks,

Josh

[nmakes]

Piggy-backing on @JBKnights question, I wonder if Linformer was intended.

From my understanding (please correct me if I'm wrong), the linearization in Linformer assumes an order in the inputs, e.g. features with (N, d) dimension are reduced to (K, d) where the N->K reduction will change if the order in the points (N,) changes. Since point clouds are sets of points and don't pertain to any ordering, the transformer layer should be order equivariant.

Thanks!