Not All Large Language Models (LLMs) Succumb to the "Reversal Curse": A
  Comparative Study of Deductive Logical Reasoning in BERT and GPT Models
Jingye Yang Da Wu Kai Wang
The "Reversal Curse" refers to the scenario where auto-regressive decoder
large language models (LLMs), such as ChatGPT, trained on "A is B" fail to
learn "B is A", demonstrating a basic failure of logical deduction. This raises
a red flag in the use of GPT models for certain general tasks such as
constructing knowledge graphs, considering their adherence to this symmetric
principle. In our study, we examined a bidirectional LLM, BERT, and found that
it is immune to the reversal curse. Driven by ongoing efforts to construct
biomedical knowledge graphs with LLMs, we also embarked on evaluating more
complex but essential deductive reasoning capabilities. This process included
first training encoder and decoder language models to master the intersection
($\cap$) and union ($\cup$) operations on two sets and then moving on to assess
their capability to infer different combinations of union ($\cup$) and
intersection ($\cap$) operations on three newly created sets. The findings
showed that while both encoder and decoder language models, trained for tasks
involving two sets (union/intersection), were proficient in such scenarios,
they encountered difficulties when dealing with operations that included three
sets (various combinations of union and intersection). Our research highlights
the distinct characteristics of encoder and decoder models in simple and
complex logical reasoning. In practice, the choice between BERT and GPT should
be guided by the specific requirements and nature of the task at hand,
leveraging their respective strengths in bidirectional context comprehension
and sequence prediction.