LLMs for Relational Reasoning: How Far are We?
Zhiming Li Yushi Cao Xiufeng Xu Junzhe Jiang Xu Liu Yon Shin Teo Shang-wei Lin Yang Liu
Large language models (LLMs) have revolutionized many areas (e.g. natural
language processing, software engineering, etc.) by achieving state-of-the-art
performance on extensive downstream tasks. Aiming to achieve robust and general
artificial intelligence, there has been a surge of interest in investigating
the reasoning ability of the LLMs. Whereas the textual and numerical reasoning
benchmarks adopted by previous works are rather shallow and simple, it is hard
to conclude that the LLMs possess strong reasoning ability by merely achieving
positive results on these benchmarks. Recent efforts have demonstrated that the
LLMs are poor at solving sequential decision-making problems that require
common-sense planning by evaluating their performance on the reinforcement
learning benchmarks. In this work, we conduct an in-depth assessment of several
state-of-the-art LLMs' reasoning ability based on the inductive logic
programming (ILP) benchmark, which is broadly recognized as a representative
and challenging measurement for evaluating logic program induction/synthesis
systems as it requires inducing strict cause-effect logic to achieve robust
deduction on independent and identically distributed (IID) and
out-of-distribution (OOD) test samples. Our evaluations illustrate that
compared with the neural program induction systems which are much smaller in
model size, the state-of-the-art LLMs are much poorer in terms of reasoning
ability by achieving much lower performance and generalization using either
natural language prompting or truth-value matrix prompting.