Unveiling Linguistic Regions in Large Language Models
Zhihao Zhang Jun Zhao Qi Zhang Tao Gui Xuanjing Huang
Large Language Models (LLMs) have demonstrated considerable cross-lingual
alignment and generalization ability. Current research primarily focuses on
improving LLMs' cross-lingual generalization capabilities. However, there is
still a lack of research on the intrinsic mechanisms of how LLMs achieve
cross-lingual alignment. From the perspective of region partitioning, this
paper conducts several investigations on the linguistic competence of LLMs. We
discover a core region in LLMs that corresponds to linguistic competence,
accounting for approximately 1% of the total model parameters. Removing this
core region by setting parameters to zero results in a significant performance
decrease across 30 different languages. Furthermore, this core region exhibits
significant dimensional dependency, perturbations to even a single parameter on
specific dimensions leading to a loss of linguistic competence. Moreover, we
discover that distinct regions exist for different monolingual families, and
disruption to these specific regions substantially reduces the LLMs'
proficiency in those corresponding languages. Our research also indicates that
freezing the core linguistic region during further pre-training can mitigate
the issue of catastrophic forgetting (CF), a common occurrence observed during
further pre-training of LLMs. Overall, exploring the LLMs' functional regions
provides insights into the foundation of their intelligence.