Case-Based or Rule-Based: How Do Transformers Do the Math?
Yi Hu Xiaojuan Tang Haotong Yang Muhan Zhang
Despite the impressive performance in a variety of complex tasks, modern
large language models (LLMs) still have trouble dealing with some math problems
that are simple and intuitive for humans, such as addition. While we can easily
learn basic rules of addition and apply them to new problems of any length,
LLMs struggle to do the same. Instead, they may rely on similar "cases" seen in
the training corpus for help. We define these two different reasoning
mechanisms as "rule-based reasoning" and "case-based reasoning". Since
rule-based reasoning is essential for acquiring the systematic generalization
ability, we aim to explore exactly whether transformers use rule-based or
case-based reasoning for math problems. Through carefully designed intervention
experiments on five math tasks, we confirm that transformers are performing
case-based reasoning, no matter whether scratchpad is used, which aligns with
the previous observations that transformers use subgraph matching/shortcut
learning to reason. To mitigate such problems, we propose a Rule-Following
Fine-Tuning (RFFT) technique to teach transformers to perform rule-based
reasoning. Specifically, we provide explicit rules in the input and then
instruct transformers to recite and follow the rules step by step. Through
RFFT, we successfully enable LLMs fine-tuned on 1-5 digit addition to
generalize to up to 12-digit addition with over 95% accuracy, which is over 40%
higher than scratchpad. The significant improvement demonstrates that teaching
LLMs to explicitly use rules helps them learn rule-based reasoning and
generalize better in length.