AD-KD: Attribution-Driven Knowledge Distillation for Language Model
  Compression
Siyue Wu Hongzhan Chen Xiaojun Quan Qifan Wang Rui Wang
Knowledge distillation has attracted a great deal of interest recently to
compress pre-trained language models. However, existing knowledge distillation
methods suffer from two limitations. First, the student model simply imitates
the teacher's behavior while ignoring the underlying reasoning. Second, these
methods usually focus on the transfer of sophisticated model-specific knowledge
but overlook data-specific knowledge. In this paper, we present a novel
attribution-driven knowledge distillation approach, which explores the
token-level rationale behind the teacher model based on Integrated Gradients
(IG) and transfers attribution knowledge to the student model. To enhance the
knowledge transfer of model reasoning and generalization, we further explore
multi-view attribution distillation on all potential decisions of the teacher.
Comprehensive experiments are conducted with BERT on the GLUE benchmark. The
experimental results demonstrate the superior performance of our approach to
several state-of-the-art methods.