Knowledge Distillation for Closed-Source Language Models
Hongzhan Chen Xiaojun Quan Hehong Chen Ming Yan Ji Zhang
Closed-source language models such as GPT-4 have achieved remarkable
performance. Many recent studies focus on enhancing the capabilities of smaller
models through knowledge distillation from closed-source language models.
However, due to the incapability to directly access the weights, hidden states,
and output distributions of these closed-source models, the distillation can
only be performed by fine-tuning smaller models with data samples generated by
closed-source language models, which constrains the effectiveness of knowledge
distillation. In this paper, we propose to estimate the output distributions of
closed-source language models within a Bayesian estimation framework, involving
both prior and posterior estimation. The prior estimation aims to derive a
prior distribution by utilizing the corpus generated by closed-source language
models, while the posterior estimation employs a proxy model to update the
prior distribution and derive a posterior distribution. By leveraging the
estimated output distribution of closed-source language models, traditional
knowledge distillation can be executed. Experimental results demonstrate that
our method surpasses the performance of current models directly fine-tuned on
data generated by closed-source language models.