Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare
  Linguistic Phenomena
Leonie Weissweiler Abdullatif Köksal Hinrich Schütze
Argument Structure Constructions (ASCs) are one of the most well-studied
construction groups, providing a unique opportunity to demonstrate the
usefulness of Construction Grammar (CxG). For example, the caused-motion
construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates
that constructions must carry meaning, otherwise the fact that ``sneeze'' in
this context causes movement cannot be explained. We form the hypothesis that
this remains challenging even for state-of-the-art Large Language Models
(LLMs), for which we devise a test based on substituting the verb with a
prototypical motion verb. To be able to perform this test at statistically
significant scale, in the absence of adequate CxG corpora, we develop a novel
pipeline of NLP-assisted collection of linguistically annotated text. We show
how dependency parsing and GPT-3.5 can be used to significantly reduce
annotation cost and thus enable the annotation of rare phenomena at scale. We
then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of
the CMC using the newly collected corpus. We find that all models struggle with
understanding the motion component that the CMC adds to a sentence.