Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling
Mahdi Karami Ali Ghodsi
In the rapidly evolving landscape of deep learning, the quest for models that
balance expressivity with computational efficiency has never been more
critical. This paper introduces Orchid, a novel architecture that reimagines
sequence modeling by incorporating a new data-dependent convolution mechanism.
Orchid is designed to address the inherent limitations of traditional attention
mechanisms, particularly their quadratic complexity, without compromising the
ability to capture long-range dependencies and in-context learning. At the core
of Orchid lies the data-dependent convolution layer, which dynamically adjusts
its kernel conditioned on input data using a dedicated conditioning neural
network. We design two simple conditioning networks that maintain shift
equivariance in the adaptive convolution operation. The dynamic nature of
data-dependent convolution kernel, coupled with gating operations, grants
Orchid high expressivity while maintaining efficiency and quasilinear
scalability for long sequences. We rigorously evaluate Orchid across multiple
domains, including language modeling and image classification, to showcase its
performance and generality. Our experiments demonstrate that Orchid
architecture not only outperforms traditional attention-based architectures
such as BERT and Vision Transformers with smaller model sizes, but also extends
the feasible sequence length beyond the limitations of the dense attention
layers. This achievement represents a significant step towards more efficient
and scalable deep learning models for sequence modeling.