Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense
  Passage Retrieval
Guangyuan Ma Xing Wu Zijia Lin Songlin Hu
Masked auto-encoder pre-training has emerged as a prevalent technique for
initializing and enhancing dense retrieval systems. It generally utilizes
additional Transformer decoder blocks to provide sustainable supervision
signals and compress contextual information into dense representations.
However, the underlying reasons for the effectiveness of such a pre-training
technique remain unclear. The usage of additional Transformer-based decoders
also incurs significant computational costs. In this study, we aim to shed
light on this issue by revealing that masked auto-encoder (MAE) pre-training
with enhanced decoding significantly improves the term coverage of input tokens
in dense representations, compared to vanilla BERT checkpoints. Building upon
this observation, we propose a modification to the traditional MAE by replacing
the decoder of a masked auto-encoder with a completely simplified Bag-of-Word
prediction task. This modification enables the efficient compression of lexical
signals into dense representations through unsupervised pre-training.
Remarkably, our proposed method achieves state-of-the-art retrieval performance
on several large-scale retrieval benchmarks without requiring any additional
parameters, which provides a 67% training speed-up compared to standard masked
auto-encoder pre-training with enhanced decoding.