Contextualized Word Vector-based Methods for Discovering Semantic
  Differences with No Training nor Word Alignment
Ryo Nagata Hiroya Takamura Naoki Otani Yoshifumi Kawasaki
In this paper, we propose methods for discovering semantic differences in
words appearing in two corpora based on the norms of contextualized word
vectors. The key idea is that the coverage of meanings is reflected in the norm
of its mean word vector. The proposed methods do not require the assumptions
concerning words and corpora for comparison that the previous methods do. All
they require are to compute the mean vector of contextualized word vectors and
its norm for each word type. Nevertheless, they are (i) robust for the skew in
corpus size; (ii) capable of detecting semantic differences in infrequent
words; and (iii) effective in pinpointing word instances that have a meaning
missing in one of the two corpora for comparison. We show these advantages for
native and non-native English corpora and also for historical corpora.