Where exactly does contextualization in a PLM happen?
Soniya Vijayakumar Tanja Bäumel Simon Ostermann Josef van Genabith
Pre-trained Language Models (PLMs) have shown to be consistently successful
in a plethora of NLP tasks due to their ability to learn contextualized
representations of words (Ethayarajh, 2019). BERT (Devlin et al., 2018), ELMo
(Peters et al., 2018) and other PLMs encode word meaning via textual context,
as opposed to static word embeddings, which encode all meanings of a word in a
single vector representation. In this work, we present a study that aims to
localize where exactly in a PLM word contextualization happens. In order to
find the location of this word meaning transformation, we investigate
representations of polysemous words in the basic BERT uncased 12 layer
architecture (Devlin et al., 2018), a masked language model trained on an
additional sentence adjacency objective, using qualitative and quantitative
measures.