Morphosyntactic probing of multilingual BERT models
Judit Acs Endre Hamerlik Roy Schwartz Noah A. Smith Andras Kornai
We introduce an extensive dataset for multilingual probing of morphological
information in language models (247 tasks across 42 languages from 10
families), each consisting of a sentence with a target word and a morphological
tag as the desired label, derived from the Universal Dependencies treebanks. We
find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features
that attain strong performance across these tasks. We then apply two methods to
locate, for each probing task, where the disambiguating information resides in
the input. The first is a new perturbation method that masks various parts of
context; the second is the classical method of Shapley values. The most
intriguing finding that emerges is a strong tendency for the preceding context
to hold more information relevant to the prediction than the following context.