Are LLMs Effective Negotiators? Systematic Evaluation of the
  Multifaceted Capabilities of LLMs in Negotiation Dialogues
Deuksin Kwon Emily Weiss Tara Kulshrestha Kushal Chawla Gale M. Lucas Jonathan Gratch
A successful negotiation demands a deep comprehension of the conversation
context, Theory-of-Mind (ToM) skills to infer the partner's motives, as well as
strategic reasoning and effective communication, making it challenging for
automated systems. Given the remarkable performance of LLMs across a variety of
NLP tasks, in this work, we aim to understand how LLMs can advance different
aspects of negotiation research, ranging from designing dialogue systems to
providing pedagogical feedback and scaling up data collection practices. To
this end, we devise a methodology to analyze the multifaceted capabilities of
LLMs across diverse dialogue scenarios covering all the time stages of a
typical negotiation interaction. Our analysis adds to the increasing evidence
for the superiority of GPT-4 across various tasks while also providing insights
into specific tasks that remain difficult for LLMs. For instance, the models
correlate poorly with human players when making subjective assessments about
the negotiation dialogues and often struggle to generate responses that are
contextually appropriate as well as strategically advantageous.