Strong hallucinations from negation and how to fix them
Nicholas Asher Swarnadeep Bhar
Despite great performance on many tasks, language models (LMs) still struggle
with reasoning, sometimes providing responses that cannot possibly be true
because they stem from logical incoherence. We call such responses
\textit{strong hallucinations} and prove that they follow from an LM's
computation of its internal representations for logical operators and outputs
from those representations. Focusing on negation, we provide a novel solution
in which negation is treated not as another element of a latent representation,
but as \textit{an operation over an LM's latent representations that constrains
how they may evolve}. We show that our approach improves model performance in
cloze prompting and natural language inference tasks with negation without
requiring training on sparse negative data.