Can a student Large Language Model perform as well as it's teacher?
Sia Gholami Marwan Omar
The burgeoning complexity of contemporary deep learning models, while
achieving unparalleled accuracy, has inadvertently introduced deployment
challenges in resource-constrained environments. Knowledge distillation, a
technique aiming to transfer knowledge from a high-capacity "teacher" model to
a streamlined "student" model, emerges as a promising solution to this dilemma.
This paper provides a comprehensive overview of the knowledge distillation
paradigm, emphasizing its foundational principles such as the utility of soft
labels and the significance of temperature scaling. Through meticulous
examination, we elucidate the critical determinants of successful distillation,
including the architecture of the student model, the caliber of the teacher,
and the delicate balance of hyperparameters. While acknowledging its profound
advantages, we also delve into the complexities and challenges inherent in the
process. Our exploration underscores knowledge distillation's potential as a
pivotal technique in optimizing the trade-off between model performance and
deployment efficiency.