Exploring the Numerical Reasoning Capabilities of Language Models: A
  Comprehensive Analysis on Tabular Data
Mubashara Akhtar Abhilash Shankarampeta Vivek Gupta Arpit Patil Oana Cocarascu Elena Simperl
Numbers are crucial for various real-world domains such as finance,
economics, and science. Thus, understanding and reasoning with numbers are
essential skills for language models to solve different tasks. While different
numerical benchmarks have been introduced in recent years, they are limited to
specific numerical aspects mostly. In this paper, we propose a hierarchical
taxonomy for numerical reasoning skills with more than ten reasoning types
across four levels: representation, number sense, manipulation, and complex
reasoning. We conduct a comprehensive evaluation of state-of-the-art models to
identify reasoning challenges specific to them. Henceforth, we develop a
diverse set of numerical probes employing a semi-automated approach. We focus
on the tabular Natural Language Inference (TNLI) task as a case study and
measure models' performance shifts. Our results show that no model consistently
excels across all numerical reasoning types. Among the probed models, FlanT5
(few-/zero-shot) and GPT-3.5 (few-shot) demonstrate strong overall numerical
reasoning skills compared to other models. Label-flipping probes indicate that
models often exploit dataset artifacts to predict the correct labels.