Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of
  Large Language Models with Misconceptions
Naiming Liu Shashank Sonkar Zichao Wang Simon Woodhead Richard G. Baraniuk
We propose novel evaluations for mathematical reasoning capabilities of Large
Language Models (LLMs) based on mathematical misconceptions. Our primary
approach is to simulate LLMs as a novice learner and an expert tutor, aiming to
identify the incorrect answer to math question resulted from a specific
misconception and to recognize the misconception(s) behind an incorrect answer,
respectively. Contrary to traditional LLMs-based mathematical evaluations that
focus on answering math questions correctly, our approach takes inspirations
from principles in educational learning sciences. We explicitly ask LLMs to
mimic a novice learner by answering questions in a specific incorrect manner
based on incomplete knowledge; and to mimic an expert tutor by identifying
misconception(s) corresponding to an incorrect answer to a question. Using
simple grade-school math problems, our experiments reveal that, while LLMs can
easily answer these questions correctly, they struggle to identify 1) the
incorrect answer corresponding to specific incomplete knowledge
(misconceptions); 2) the misconceptions that explain particular incorrect
answers. Our study indicates new opportunities for enhancing LLMs' math
reasoning capabilities, especially on developing robust student simulation and
expert tutoring models in the educational applications such as intelligent
tutoring systems.