Functional Benchmarks for Robust Evaluation of Reasoning Performance,
  and the Reasoning Gap
Saurabh Srivastava Annarose M B Anto P V Shashank Menon Ajay Sukumar Adwaith Samod T Alan Philipose Stevin Prince Sooraj Thomas
We propose a framework for robust evaluation of reasoning capabilities of
language models, using functional variants of benchmarks. Models that solve a
reasoning test should exhibit no difference in performance over the static
version of a problem compared to a snapshot of the functional variant. We have
rewritten the relevant fragment of the MATH benchmark into its functional
variant MATH(), with functionalization of other benchmarks to follow. When
evaluating current state-of-the-art models over snapshots of MATH(), we find a
reasoning gap -- the percentage difference between the static and functional
accuracies. We find reasoning gaps from 58.35% to 80.31% among the
state-of-the-art closed and open weights models that perform well on static
benchmarks, with the caveat that the gaps are likely to be smaller with more
sophisticated prompting strategies. Here we show that models which anecdotally
have good reasoning performance over real-world tasks, have quantifiable lower
gaps, motivating the open problem of building "gap 0" models. Code for
evaluation and new evaluation datasets, three MATH() snapshots, are publicly
available at https://github.com/consequentai/fneval/.