MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding
  Length Extrapolation
Weiguo Gao
When the predicted sequence length exceeds the length seen during training,
the transformer's inference accuracy diminishes. Existing relative position
encoding methods, such as those based on the ALiBi technique, address the
length extrapolation challenge exclusively through the implementation of a
single kernel function, which introduces a constant bias to every post-softmax
attention scores according to their distance. These approaches do not
investigate or employ multiple kernel functions to address the extrapolation
challenge. Drawing on the ALiBi approach, this study proposes a novel relative
positional encoding method, called MEP, which employs a weighted average to
combine distinct kernel functions(such as the exponential kernel and the
Gaussian kernel) to generate a bias that is applied to post-softmax attention
scores. Initially, the framework utilizes various kernel functions to construct
multiple kernel functions. Each kernel function adheres to a consistent mean
weight coefficient, harnessing the synergistic advantages of different kernels
to formulate an innovative bias function. Subsequently, specific slopes are
tailored for each kernel function, applying penalties at varying rates, to
enhance the model's extrapolation capabilities. Finally, this bias is
seamlessly incorporated as a penalty to the post-softmax scores. We present two
distinct versions of our method: a parameter-free variant that requires no new
learnable parameters, which enhances length extrapolation capabilities without
compromising training efficiency, and a parameterized variant capable of
integrating state-of-the-art techniques. Empirical evaluations across diverse
datasets have demonstrated that both variants of our method achieve
state-of-the-art performance, outperforming traditional parameter-free and
parameterized approaches.