Using Counterfactual Tasks to Evaluate the Generality of Analogical
  Reasoning in Large Language Models
Martha Lewis Melanie Mitchell
Large language models (LLMs) have performed well on several reasoning
benchmarks, including ones that test analogical reasoning abilities. However,
it has been debated whether they are actually performing humanlike abstract
reasoning or instead employing less general processes that rely on similarity
to what has been seen in their training data. Here we investigate the
generality of analogy-making abilities previously claimed for LLMs (Webb,
Holyoak, & Lu, 2023). We take one set of analogy problems used to evaluate LLMs
and create a set of "counterfactual" variants-versions that test the same
abstract reasoning abilities but that are likely dissimilar from any
pre-training data. We test humans and three GPT models on both the original and
counterfactual problems, and show that, while the performance of humans remains
high for all the problems, the GPT models' performance declines sharply on the
counterfactual set. This work provides evidence that, despite previously
reported successes of LLMs on analogical reasoning, these models lack the
robustness and generality of human analogy-making.