Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
Ivan Sekulić Silvia Terragni Victor Guimarães Nghia Khau Bruna Guedes Modestas Filipavicius André Ferreira Manso Roland Mathis
In the realm of dialogue systems, user simulation techniques have emerged as
a game-changer, redefining the evaluation and enhancement of task-oriented
dialogue (TOD) systems. These methods are crucial for replicating real user
interactions, enabling applications like synthetic data augmentation, error
detection, and robust evaluation. However, existing approaches often rely on
rigid rule-based methods or on annotated data. This paper introduces DAUS, a
Domain-Aware User Simulator. Leveraging large language models, we fine-tune
DAUS on real examples of task-oriented dialogues. Results on two relevant
benchmarks showcase significant improvements in terms of user goal fulfillment.
Notably, we have observed that fine-tuning enhances the simulator's coherence
with user goals, effectively mitigating hallucinations -- a major source of
inconsistencies in simulator responses.