Bidirectional Long-Range Parser for Sequential Data Understanding
George Leotescu Daniel Voinea Alin-Ionut Popa
The transformer is a powerful data modelling framework responsible for
remarkable performance on a wide range of tasks. However, they are limited in
terms of scalability as it is suboptimal and inefficient to process
long-sequence data. To this purpose we introduce BLRP (Bidirectional Long-Range
Parser), a novel and versatile attention mechanism designed to increase
performance and efficiency on long-sequence tasks. It leverages short and long
range heuristics in the form of a local sliding window approach combined with a
global bidirectional latent space synthesis technique. We show the benefits and
versatility of our approach on vision and language domains by demonstrating
competitive results against state-of-the-art methods on the Long-Range-Arena
and CIFAR benchmarks together with ablations demonstrating the computational
efficiency.