A Frustratingly Easy Improvement for Position Embeddings via Random
  Padding
Mingxu Tao Yansong Feng Dongyan Zhao
Position embeddings, encoding the positional relationships among tokens in
text sequences, make great contributions to modeling local context features in
Transformer-based pre-trained language models. However, in Extractive Question
Answering, position embeddings trained with instances of varied context lengths
may not perform well as we expect. Since the embeddings of rear positions are
updated fewer times than the front position embeddings, the rear ones may not
be properly trained. In this paper, we propose a simple but effective strategy,
Random Padding, without any modifications to architectures of existing
pre-trained language models. We adjust the token order of input sequences when
fine-tuning, to balance the number of updating times of every position
embedding. Experiments show that Random Padding can significantly improve model
performance on the instances whose answers are located at rear positions,
especially when models are trained on short contexts but evaluated on long
contexts. Our code and data will be released for future research.