A Comparative Analysis of Task-Agnostic Distillation Methods for
  Compressing Transformer Language Models
Takuma Udagawa Aashka Trivedi Michele Merler Bishwaranjan Bhattacharjee
Large language models have become a vital component in modern NLP, achieving
state of the art performance in a variety of tasks. However, they are often
inefficient for real-world deployment due to their expensive inference costs.
Knowledge distillation is a promising technique to improve their efficiency
while retaining most of their effectiveness. In this paper, we reproduce,
compare and analyze several representative methods for task-agnostic
(general-purpose) distillation of Transformer language models. Our target of
study includes Output Distribution (OD) transfer, Hidden State (HS) transfer
with various layer mapping strategies, and Multi-Head Attention (MHA) transfer
based on MiniLMv2. Through our extensive experiments, we study the
effectiveness of each method for various student architectures in both
monolingual (English) and multilingual settings. Overall, we show that MHA
transfer based on MiniLMv2 is generally the best option for distillation and
explain the potential reasons behind its success. Moreover, we show that HS
transfer remains as a competitive baseline, especially under a sophisticated
layer mapping strategy, while OD transfer consistently lags behind other
approaches. Findings from this study helped us deploy efficient yet effective
student models for latency-critical applications.