f-Divergence Minimization for Sequence-Level Knowledge Distillation
Yuqiao Wen Zichao Li Wenyu Du Lili Mou
Knowledge distillation (KD) is the process of transferring knowledge from a
large model to a small one. It has gained increasing attention in the natural
language processing community, driven by the demands of compressing
ever-growing language models. In this work, we propose an f-DISTILL framework,
which formulates sequence-level knowledge distillation as minimizing a
generalized f-divergence function. We propose four distilling variants under
our framework and show that existing SeqKD and ENGINE approaches are
approximations of our f-DISTILL methods. We further derive step-wise
decomposition for our f-DISTILL, reducing intractable sequence-level divergence
to word-level losses that can be computed in a tractable manner. Experiments
across four datasets show that our methods outperform existing KD approaches,
and that our symmetric distilling losses can better force the student to learn
from the teacher distribution.