What does the Failure to Reason with "Respectively" in Zero/Few-Shot
  Settings Tell Us about Language Models?
Ruixiang Cui Seolhwa Lee Daniel Hershcovich Anders Søgaard
Humans can effortlessly understand the coordinate structure of sentences such
as "Niels Bohr and Kurt Cobain were born in Copenhagen and Seattle,
respectively". In the context of natural language inference (NLI), we examine
how language models (LMs) reason with respective readings (Gawron and Kehler,
2004) from two perspectives: syntactic-semantic and commonsense-world
knowledge. We propose a controlled synthetic dataset WikiResNLI and a naturally
occurring dataset NatResNLI to encompass various explicit and implicit
realizations of "respectively". We show that fine-tuned NLI models struggle
with understanding such readings without explicit supervision. While few-shot
learning is easy in the presence of explicit cues, longer training is required
when the reading is evoked implicitly, leaving models to rely on common sense
inferences. Furthermore, our fine-grained analysis indicates models fail to
generalize across different constructions. To conclude, we demonstrate that LMs
still lag behind humans in generalizing to the long tail of linguistic
constructions.