Analysis of the Reasoning with Redundant Information Provided Ability of
  Large Language Models
Wenbei Xie
Recent advancements in Large Language Models (LLMs) have demonstrated
impressive capabilities across a range of natural language processing tasks,
especially in reasoning, a cornerstone for achieving Artificial General
Intelligence (AGI). However, commonly used benchmarks may not fully encapsulate
the inferential abilities of these models in real-world scenarios. To address
this gap, a new form of Question-Answering (QA) task, termed Reasoning with
Redundant Information Provided (RRIP), is introduced. The study designed a
modified version of the grade school math 8K (GSM-8K) dataset which has several
variants focusing on different attributes of redundant information. This
investigation evaluates two popular LLMs, LlaMA2-13B-chat and generative
pre-trained transformer 3.5 (GPT-3.5), contrasting their performance on
traditional QA tasks against the RRIP tasks. Findings indicate that while these
models achieved moderate success on standard QA benchmarks, their performance
notably declines when assessed on RRIP tasks. The study not only highlights the
limitations of current LLMs in handling redundant information but also suggests
that future training of these models should focus on incorporating redundant
information into the training data to increase the performance on RRIP tasks.