Position Interpolation Improves ALiBi Extrapolation
Faisal Al-Khateeb Nolan Dey Daria Soboleva Joel Hestness
Linear position interpolation helps pre-trained models using rotary position
embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using
linear position interpolation to extend the extrapolation range of models using
Attention with Linear Biases (ALiBi). We find position interpolation
significantly improves extrapolation capability on upstream language modelling
and downstream summarization and retrieval tasks.