A Joint Matrix Factorization Analysis of Multilingual Representations
Zheng Zhao Yftah Ziser Bonnie Webber Shay B. Cohen
We present an analysis tool based on joint matrix factorization for comparing
latent representations of multilingual and monolingual models. An alternative
to probing, this tool allows us to analyze multiple sets of representations in
a joint manner. Using this tool, we study to what extent and how
morphosyntactic features are reflected in the representations learned by
multilingual pre-trained models. We conduct a large-scale empirical study of
over 33 languages and 17 morphosyntactic categories. Our findings demonstrate
variations in the encoding of morphosyntactic information across upper and
lower layers, with category-specific differences influenced by language
properties. Hierarchical clustering of the factorization outputs yields a tree
structure that is related to phylogenetic trees manually crafted by linguists.
Moreover, we find the factorization outputs exhibit strong associations with
performance observed across different cross-lingual tasks. We release our code
to facilitate future research.