Practical Insights into Knowledge Distillation for Pre-Trained Models
Norah Alballa Marco Canini
This research investigates the enhancement of knowledge distillation (KD)
processes in pre-trained models, an emerging field in knowledge transfer with
significant implications for distributed training and federated learning
environments. These environments benefit from reduced communication demands and
accommodate various model architectures. Despite the adoption of numerous KD
approaches for transferring knowledge among pre-trained models, a comprehensive
understanding of KD's application in these scenarios is lacking. Our study
conducts an extensive comparison of multiple KD techniques, including standard
KD, tuned KD (via optimized temperature and weight parameters), deep mutual
learning, and data partitioning KD. We assess these methods across various data
distribution strategies to identify the most effective contexts for each.
Through detailed examination of hyperparameter tuning, informed by extensive
grid search evaluations, we pinpoint when adjustments are crucial to enhance
model performance. This paper sheds light on optimal hyperparameter settings
for distinct data partitioning scenarios and investigates KD's role in
improving federated learning by minimizing communication rounds and expediting
the training process. By filling a notable void in current research, our
findings serve as a practical framework for leveraging KD in pre-trained models
within collaborative and federated learning frameworks.