Dynamic Corrective Self-Distillation for Better Fine-Tuning of
  Pretrained Models
Ibtihel Amara Vinija Jain Aman Chadha
We tackle the challenging issue of aggressive fine-tuning encountered during
the process of transfer learning of pre-trained language models (PLMs) with
limited labeled downstream data. This problem primarily results in a decline in
performance on the subsequent task. Inspired by the adaptive boosting method in
traditional machine learning, we present an effective dynamic corrective
self-distillation (DCS) approach to improve the fine-tuning of the PLMs. Our
technique involves performing a self-distillation mechanism where, at each
iteration, the student model actively adapts and corrects itself by dynamically
adjusting the weights assigned to individual data points. This iterative
self-correcting process significantly enhances the overall fine-tuning
capability of PLMs, leading to improved performance and robustness. We
conducted comprehensive evaluations using the GLUE benchmark demonstrating the
efficacy of our method in enhancing the fine-tuning process for various PLMs
across diverse downstream tasks.