Semantics of Multiword Expressions in Transformer-Based Models: A Survey
Filip Miletić Sabine Schulte im Walde
Multiword expressions (MWEs) are composed of multiple words and exhibit
variable degrees of compositionality. As such, their meanings are notoriously
difficult to model, and it is unclear to what extent this issue affects
transformer architectures. Addressing this gap, we provide the first in-depth
survey of MWE processing with transformer models. We overall find that they
capture MWE semantics inconsistently, as shown by reliance on surface patterns
and memorized information. MWE meaning is also strongly localized,
predominantly in early layers of the architecture. Representations benefit from
specific linguistic properties, such as lower semantic idiosyncrasy and
ambiguity of target expressions. Our findings overall question the ability of
transformer models to robustly capture fine-grained semantics. Furthermore, we
highlight the need for more directly comparable evaluation setups.