Neuron Specialization: Leveraging intrinsic task modularity for
  multilingual machine translation
Shaomu Tan Di Wu Christof Monz
Training a unified multilingual model promotes knowledge transfer but
inevitably introduces negative interference. Language-specific modeling methods
show promise in reducing interference. However, they often rely on heuristics
to distribute capacity and struggle to foster cross-lingual transfer via
isolated modules. In this paper, we explore intrinsic task modularity within
multilingual networks and leverage these observations to circumvent
interference under multilingual translation. We show that neurons in the
feed-forward layers tend to be activated in a language-specific manner.
Meanwhile, these specialized neurons exhibit structural overlaps that reflect
language proximity, which progress across layers. Based on these findings, we
propose Neuron Specialization, an approach that identifies specialized neurons
to modularize feed-forward layers and then continuously updates them through
sparse networks. Extensive experiments show that our approach achieves
consistent performance gains over strong baselines with additional analyses
demonstrating reduced interference and increased knowledge transfer.