Investigating Content Planning for Navigating Trade-offs in
  Knowledge-Grounded Dialogue
Kushal Chawla Hannah Rashkin Gaurav Singh Tomar David Reitter
Knowledge-grounded dialogue generation is a challenging task because it
requires satisfying two fundamental yet often competing constraints: being
responsive in a manner that is specific to what the conversation partner has
said while also being attributable to an underlying source document. In this
work, we bring this trade-off between these two objectives (specificity and
attribution) to light and ask the question: Can explicit content planning
before the response generation help the model to address this challenge? To
answer this question, we design a framework called PLEDGE, which allows us to
experiment with various plan variables explored in prior work, supporting both
metric-agnostic and metric-aware approaches. While content planning shows
promise, our results on whether it can actually help to navigate this trade-off
are mixed -- planning mechanisms that are metric-aware (use automatic metrics
during training) are better at automatic evaluations but underperform in human
judgment compared to metric-agnostic mechanisms. We discuss how this may be
caused by over-fitting to automatic metrics and the need for future work to
better calibrate these metrics towards human judgment. We hope the observations
from our analysis will inform future work that aims to apply content planning
in this context.