What is Lost in Knowledge Distillation?
Manas Mohanty Tanya Roosta Peyman Passban
Deep neural networks (DNNs) have improved NLP tasks significantly, but
training and maintaining such networks could be costly. Model compression
techniques, such as, knowledge distillation (KD), have been proposed to address
the issue; however, the compression process could be lossy. Motivated by this,
our work investigates how a distilled student model differs from its teacher,
if the distillation process causes any information losses, and if the loss
follows a specific pattern. Our experiments aim to shed light on the type of
tasks might be less or more sensitive to KD by reporting data points on the
contribution of different factors, such as the number of layers or attention
heads. Results such as ours could be utilized when determining effective and
efficient configurations to achieve optimal information transfers between
larger (teacher) and smaller (student) models.