Perceiving Longer Sequences With Bi-Directional Cross-Attention
  Transformers
Markus Hiller Krista A. Ehinger Tom Drummond
We present a novel bi-directional Transformer architecture (BiXT) which
scales linearly with input size in terms of computational cost and memory
consumption, but does not suffer the drop in performance or limitation to only
one input modality seen with other efficient Transformer-based approaches. BiXT
is inspired by the Perceiver architectures but replaces iterative attention
with an efficient bi-directional cross-attention module in which input tokens
and latent variables attend to each other simultaneously, leveraging a
naturally emerging attention-symmetry between the two. This approach unlocks a
key bottleneck experienced by Perceiver-like architectures and enables the
processing and interpretation of both semantics (`what') and location (`where')
to develop alongside each other over multiple layers -- allowing its direct
application to dense and instance-based tasks alike. By combining efficiency
with the generality and performance of a full Transformer architecture, BiXT
can process longer sequences like point clouds or images at higher feature
resolutions and achieves competitive performance across a range of tasks like
point cloud part segmentation, semantic image segmentation and image
classification.