PRODIGy: a PROfile-based DIalogue Generation dataset
Daniela Occhipinti Serra Sinem Tekiroglu Marco Guerini
Providing dialogue agents with a profile representation can improve their
consistency and coherence, leading to better conversations. However, current
profile-based dialogue datasets for training such agents contain either
explicit profile representations that are simple and dialogue-specific, or
implicit representations that are difficult to collect. In this work, we
propose a unified framework in which we bring together both standard and more
sophisticated profile representations by creating a new resource where each
dialogue is aligned with all possible speaker representations such as
communication style, biographies, and personality. This framework allows to
test several baselines built using generative language models with several
profile configurations. The automatic evaluation shows that profile-based
models have better generalisation capabilities than models trained on dialogues
only, both in-domain and cross-domain settings. These results are consistent
for fine-tuned models and instruction-based LLMs. Additionally, human
evaluation demonstrates a clear preference for generations consistent with both
profile and context. Finally, to account for possible privacy concerns, all
experiments are done under two configurations: inter-character and
intra-character. In the former, the LM stores the information about the
character in its internal representation, while in the latter, the LM does not
retain any personal information but uses it only at inference time.