Unveiling Multilinguality in Transformer Models: Exploring Language
  Specificity in Feed-Forward Networks
Sunit Bhattacharya Ondrej Bojar
Recent research suggests that the feed-forward module within Transformers can
be viewed as a collection of key-value memories, where the keys learn to
capture specific patterns from the input based on the training examples. The
values then combine the output from the 'memories' of the keys to generate
predictions about the next token. This leads to an incremental process of
prediction that gradually converges towards the final token choice near the
output layers. This interesting perspective raises questions about how
multilingual models might leverage this mechanism. Specifically, for
autoregressive models trained on two or more languages, do all neurons (across
layers) respond equally to all languages? No! Our hypothesis centers around the
notion that during pretraining, certain model parameters learn strong
language-specific features, while others learn more language-agnostic (shared
across languages) features. To validate this, we conduct experiments utilizing
parallel corpora of two languages that the model was initially pretrained on.
Our findings reveal that the layers closest to the network's input or output
tend to exhibit more language-specific behaviour compared to the layers in the
middle.