Data-Free Distillation of Language Model by Text-to-Text Transfer
Zheyuan Bai Xinduo Liu Hailin Hu Tianyu Guo Qinghua Zhang Yunhe Wang
Data-Free Knowledge Distillation (DFKD) plays a vital role in compressing the
model when original training data is unavailable. Previous works for DFKD in
NLP mainly focus on distilling encoder-only structures like BERT on
classification tasks, which overlook the notable progress of generative
language modeling. In this work, we propose a novel DFKD framework, namely
DFKD-T$^{3}$, where the pretrained generative language model can also serve as
a controllable data generator for model compression. This novel framework
DFKD-T$^{3}$ leads to an end-to-end learnable text-to-text framework to
transform the general domain corpus to compression-friendly task data,
targeting to improve both the \textit{specificity} and \textit{diversity}.
Extensive experiments show that our method can boost the distillation
performance in various downstream tasks such as sentiment analysis, linguistic
acceptability, and information extraction. Furthermore, we show that the
generated texts can be directly used for distilling other language models and
outperform the SOTA methods, making our method more appealing in a general DFKD
setting. Our code is available at
https://gitee.com/mindspore/models/tree/master/research/nlp/DFKD\_T3.