Towards the Law of Capacity Gap in Distilling Language Models
Chen Zhang Dawei Song Zheyu Ye Yan Gao
Language model (LM) distillation is a trending area that aims to distil the
knowledge resided in a large teacher LM to a small student one. While various
methods have been proposed to push the distillation to its limits, it is still
a pain distilling LMs when a large capacity gap is exhibited between the
teacher and the student LMs. The pain is mainly resulted by the curse of
capacity gap, which describes that a larger teacher LM cannot always lead to a
better student LM than one distilled from a smaller teacher LM due to the
affect of capacity gap increment. That is, there is likely an optimal point
yielding the best student LM along the scaling course of the teacher LM. Even
worse, the curse of capacity gap can be only partly yet not fully lifted as
indicated in previous studies.
  However, the tale is not ever one-sided. Although a larger teacher LM has
better performance than a smaller teacher LM, it is much more
resource-demanding especially in the context of recent large LMs (LLMs).
Consequently, instead of sticking to lifting the curse, leaving the curse as is
should be arguably fine. Even better, in this paper, we reveal that the optimal
capacity gap is almost consistent across different student scales and
architectures, fortunately turning the curse into the law of capacity gap. The
law later guides us to distil a 3B student LM (termed MiniMA) from a 7B teacher
LM (adapted LLaMA2-7B). MiniMA is demonstrated to yield a new
compute-performance pareto frontier among existing 3B LMs on commonly used
benchmarks, and its instruction-tuned version (termed MiniChat) outperforms a
wide range of 3B competitors in GPT4 evaluation and could even compete with
several 7B chat models.