Confidence Preservation Property in Knowledge Distillation Abstractions
Dmitry Vengertsev Elena Sherman
Social media platforms prevent malicious activities by detecting harmful
content of posts and comments. To that end, they employ large-scale deep neural
network language models for sentiment analysis and content understanding. Some
models, like BERT, are complex, and have numerous parameters, which makes them
expensive to operate and maintain. To overcome these deficiencies, industry
experts employ a knowledge distillation compression technique, where a
distilled model is trained to reproduce the classification behavior of the
original model. The distillation processes terminates when the distillation
loss function reaches the stopping criteria. This function is mainly designed
to ensure that the original and the distilled models exhibit alike
classification behaviors. However, besides classification accuracy, there are
additional properties of the original model that the distilled model should
preserve to be considered as an appropriate abstraction. In this work, we
explore whether distilled TinyBERT models preserve confidence values of the
original BERT models, and investigate how this confidence preservation property
could guide tuning hyperparameters of the distillation process.