Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from
  a Parametric Perspective
Ming Zhong Chenxin An Weizhu Chen Jiawei Han Pengcheng He
Large Language Models (LLMs) inherently encode a wealth of knowledge within
their parameters through pre-training on extensive corpora. While prior
research has delved into operations on these parameters to manipulate the
underlying implicit knowledge (encompassing detection, editing, and merging),
there remains an ambiguous understanding regarding their transferability across
models with varying scales. In this paper, we seek to empirically investigate
knowledge transfer from larger to smaller models through a parametric
perspective. To achieve this, we employ sensitivity-based techniques to extract
and align knowledge-specific parameters between different LLMs. Moreover, the
LoRA module is used as the intermediary mechanism for injecting the extracted
knowledge into smaller models. Evaluations across four benchmarks validate the
efficacy of our proposed method. Our findings highlight the critical factors
contributing to the process of parametric knowledge transfer, underscoring the
transferability of model parameters across LLMs of different scales. We release
code and data at \url{https://github.com/maszhongming/ParaKnowTransfer}.