PromptKD: Distilling Student-Friendly Knowledge for Generative Language
  Models via Prompt Tuning
Gyeongman Kim Doohyuk Jang Eunho Yang
Recent advancements in large language models (LLMs) have raised concerns
about inference costs, increasing the need for research into model compression.
While knowledge distillation (KD) is a prominent method for this, research on
KD for generative language models like LLMs is relatively sparse, and the
approach of distilling student-friendly knowledge, which has shown promising
performance in KD for classification models, remains unexplored in generative
language models. To explore this approach, we propose PromptKD, a simple yet
effective method that utilizes prompt tuning - for the first time in KD - to
enable generative language models to transfer student-friendly knowledge.
Unlike previous works in classification that require fine-tuning the entire
teacher model for extracting student-friendly knowledge, PromptKD achieves
similar effects by adding a small number of prompt tokens and tuning only the
prompt with student guidance. Extensive experiments on instruction-following
datasets using the GPT-2 model family show that PromptKD achieves
state-of-the-art performance while adding only 0.0007% of the teacher's
parameters as prompts. Further analysis suggests that distilling
student-friendly knowledge alleviates exposure bias effectively throughout the
entire training process, leading to performance enhancements.