This is not a Dataset: A Large Negation Benchmark to Challenge Large
  Language Models
Iker García-Ferrero Begoña Altuna Javier Álvez Itziar Gonzalez-Dios German Rigau
Although large language models (LLMs) have apparently acquired a certain
level of grammatical knowledge and the ability to make generalizations, they
fail to interpret negation, a crucial step in Natural Language Processing. We
try to clarify the reasons for the sub-optimal performance of LLMs
understanding negation. We introduce a large semi-automatically generated
dataset of circa 400,000 descriptive sentences about commonsense knowledge that
can be true or false in which negation is present in about 2/3 of the corpus in
different forms. We have used our dataset with the largest available open LLMs
in a zero-shot approach to grasp their generalization and inference capability
and we have also fine-tuned some of the models to assess whether the
understanding of negation can be trained. Our findings show that, while LLMs
are proficient at classifying affirmative sentences, they struggle with
negative sentences and lack a deep understanding of negation, often relying on
superficial cues. Although fine-tuning the models on negative sentences
improves their performance, the lack of generalization in handling negation is
persistent, highlighting the ongoing challenges of LLMs regarding negation
understanding and generalization. The dataset and code are publicly available.