Text Representation Distillation via Information Bottleneck Principle
Yanzhao Zhang Dingkun Long Zehan Li Pengjun Xie
Pre-trained language models (PLMs) have recently shown great success in text
representation field. However, the high computational cost and high-dimensional
representation of PLMs pose significant challenges for practical applications.
To make models more accessible, an effective method is to distill large models
into smaller representation models. In order to relieve the issue of
performance degradation after distillation, we propose a novel Knowledge
Distillation method called IBKD. This approach is motivated by the Information
Bottleneck principle and aims to maximize the mutual information between the
final representation of the teacher and student model, while simultaneously
reducing the mutual information between the student model's representation and
the input data. This enables the student model to preserve important learned
information while avoiding unnecessary information, thus reducing the risk of
over-fitting. Empirical studies on two main downstream applications of text
representation (Semantic Textual Similarity and Dense Retrieval tasks)
demonstrate the effectiveness of our proposed approach.