Synthetic Dialogue Dataset Generation using LLM Agents
Yelaman Abdullin Diego Molla-Aliod Bahadorreza Ofoghi John Yearwood Qingyang Li
Linear programming (LP) problems are pervasive in real-life applications.
However, despite their apparent simplicity, an untrained user may find it
difficult to determine the linear model of their specific problem. We envisage
the creation of a goal-oriented conversational agent that will engage in
conversation with the user to elicit all information required so that a
subsequent agent can generate the linear model. In this paper, we present an
approach for the generation of sample dialogues that can be used to develop and
train such a conversational agent. Using prompt engineering, we develop two
agents that "talk" to each other, one acting as the conversational agent, and
the other acting as the user. Using a set of text descriptions of linear
problems from NL4Opt available to the user only, the agent and the user engage
in conversation until the agent has retrieved all key information from the
original problem description. We also propose an extrinsic evaluation of the
dialogues by assessing how well the summaries generated by the dialogues match
the original problem descriptions. We conduct human and automatic evaluations,
including an evaluation approach that uses GPT-4 to mimic the human evaluation
metrics. The evaluation results show an overall good quality of the dialogues,
though research is still needed to improve the quality of the GPT-4 evaluation
metrics. The resulting dialogues, including the human annotations of a subset,
are available to the research community. The conversational agent used for the
generation of the dialogues can be used as a baseline.