I Wish to Have an Argument: Argumentative Reasoning in Large Language
  Models
Adrian de Wynter Tommy Yuan
We evaluate the ability of contemporary large language models (LLMs) to
perform argumentative reasoning. We frame our experiments in terms of the
argument mining (AM) and argument pair extraction (APE) tasks, and evaluate
their ability to perform reasoning at increasing levels of abstraction in the
input and output representations (e.g., arbitrary label sets, semantic graphs).
We find that, although LLMs are able to match or surpass the state-of-the-art
in AM and APE, their argumentative reasoning performance is very dependent on
the input and output representation. We also find an "exemplar effect", where
too many exemplars increasingly become detrimental for task performance, and
about 4-5 being the optimal amount. Neither result extends to chain-of-thought
(CoT) prompting: we find the exemplar effect to be nullified, and our results
suggest that CoT allows for better performance under ill-conditioned problems.
We hope that the work reported contributes to the improvement of argumentative
reasoning in LLMs.