Unraveling Babel: Exploring Multilingual Activation Patterns within
  Large Language Models
Weize Liu Yinlong Xu Hongxia Xu Jintai Chen Xuming Hu Jian Wu
Recently, large language models (LLMs) have achieved tremendous breakthroughs
in the field of language processing, yet their mechanisms in processing
multiple languages remain agnostic. Therefore, in this work we study the
multilingual activation patterns of LLMs. By transforming the original Large
Language Models (LLMs) into a Mixture of Experts (MoE) architecture, we analyze
the expert activation patterns when processing various languages and
demonstrate the connections of these activation patterns at the level of
language families. We discover the existence of non-language-specific neurons
as well as language-specific activation neurons. Further exploration even
showcases that merely leveraging high-frequency activation neurons can
accelerate inference while maintaining comparable performance. These findings
shed light on the LLMs' multilingual processing mechanism, and are of
significant importance in guiding the multilingual training and model pruning
of LLMs.