DPBERT: Efficient Inference for BERT based on Dynamic Planning
Weixin Wu Hankz Hankui Zhuo
Large-scale pre-trained language models such as BERT have contributed
significantly to the development of NLP. However, those models require large
computational resources, making it difficult to be applied to mobile devices
where computing power is limited. In this paper we aim to address the weakness
of existing input-adaptive inference methods which fail to take full advantage
of the structure of BERT. We propose Dynamic Planning in BERT, a novel
fine-tuning strategy that can accelerate the inference process of BERT through
selecting a subsequence of transformer layers list of backbone as a
computational path for an input sample. To do this, our approach adds a
planning module to the original BERT model to determine whether a layer is
included or bypassed during inference. Experimental results on the GLUE
benchmark exhibit that our method reduces latency to 75\% while maintaining
98\% accuracy, yielding a better accuracy-speed trade-off compared to
state-of-the-art input-adaptive methods.