NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
Eli Schwartz Leshem Choshen Joseph Shtok Sivan Doveh Leonid Karlinsky Assaf Arbelle
Language models struggle with handling numerical data and performing
arithmetic operations. We hypothesize that this limitation can be partially
attributed to non-intuitive textual numbers representation. When a digit is
read or generated by a causal language model it does not know its place value
(e.g. thousands vs. hundreds) until the entire number is processed. To address
this issue, we propose a simple adjustment to how numbers are represented by
including the count of digits before each number. For instance, instead of
"42", we suggest using "{2:42}" as the new format. This approach, which we term
NumeroLogic, offers an added advantage in number generation by serving as a
Chain of Thought (CoT). By requiring the model to consider the number of digits
first, it enhances the reasoning process before generating the actual number.
We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic
formatting. We further demonstrate NumeroLogic applicability to general natural
language modeling, improving language understanding performance in the MMLU
benchmark.