Language models are not naysayers: An analysis of language models on
  negation benchmarks
Thinh Hung Truong Timothy Baldwin Karin Verspoor Trevor Cohn
Negation has been shown to be a major bottleneck for masked language models,
such as BERT. However, whether this finding still holds for larger-sized
auto-regressive language models (``LLMs'') has not been studied
comprehensively. With the ever-increasing volume of research and applications
of LLMs, we take a step back to evaluate the ability of current-generation LLMs
to handle negation, a fundamental linguistic phenomenon that is central to
language understanding. We evaluate different LLMs -- including the open-source
GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks.
Through systematic experimentation with varying model sizes and prompts, we
show that LLMs have several limitations including insensitivity to the presence
of negation, an inability to capture the lexical semantics of negation, and a
failure to reason under negation.