torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free
  Deep Learning Studies: A Case Study on NLP
Yoshitomo Matsubara
Reproducibility in scientific work has been becoming increasingly important
in research communities such as machine learning, natural language processing,
and computer vision communities due to the rapid development of the research
domains supported by recent advances in deep learning. In this work, we present
a significantly upgraded version of torchdistill, a modular-driven coding-free
deep learning framework significantly upgraded from the initial release, which
supports only image classification and object detection tasks for reproducible
knowledge distillation experiments. To demonstrate that the upgraded framework
can support more tasks with third-party libraries, we reproduce the GLUE
benchmark results of BERT models using a script based on the upgraded
torchdistill, harmonizing with various Hugging Face libraries. All the 27
fine-tuned BERT models and configurations to reproduce the results are
published at Hugging Face, and the model weights have already been widely used
in research communities. We also reimplement popular small-sized models and new
knowledge distillation methods and perform additional experiments for computer
vision tasks.