Improving Knowledge Distillation for BERT Models: Loss Functions,
  Mapping Methods, and Weight Tuning
Apoorv Dankar Adeem Jassani Kartikaeya Kumar
The use of large transformer-based models such as BERT, GPT, and T5 has led
to significant advancements in natural language processing. However, these
models are computationally expensive, necessitating model compression
techniques that reduce their size and complexity while maintaining accuracy.
This project investigates and applies knowledge distillation for BERT model
compression, specifically focusing on the TinyBERT student model. We explore
various techniques to improve knowledge distillation, including experimentation
with loss functions, transformer layer mapping methods, and tuning the weights
of attention and representation loss and evaluate our proposed techniques on a
selection of downstream tasks from the GLUE benchmark. The goal of this work is
to improve the efficiency and effectiveness of knowledge distillation, enabling
the development of more efficient and accurate models for a range of natural
language processing tasks.