The Da Vinci Code of Large Pre-trained Language Models: Deciphering
  Degenerate Knowledge Neurons
Yuheng Chen Pengfei Cao Yubo Chen Yining Wang Shengping Liu Kang Liu Jun Zhao
This study explores the mechanism of factual knowledge storage in pre-trained
language models (PLMs). Previous research suggests that factual knowledge is
stored within multi-layer perceptron weights, and some storage units exhibit
degeneracy, referred to as Degenerate Knowledge Neurons (DKNs). This paper
provides a comprehensive definition of DKNs that covers both structural and
functional aspects, pioneering the study of structures in PLMs' factual
knowledge storage units. Based on this, we introduce the Neurological Topology
Clustering method, which allows the formation of DKNs in any numbers and
structures, leading to a more accurate DKN acquisition. Furthermore, we
introduce the Neuro-Degeneracy Analytic Analysis Framework, which uniquely
integrates model robustness, evolvability, and complexity for a holistic
assessment of PLMs. Within this framework, our execution of 34 experiments
across 2 PLMs, 4 datasets, and 6 settings highlights the critical role of DKNs.
The code will be available soon.