The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in
  Open-domain Conversational Question Answering
Sabrina Chiesurin Dimitris Dimakopoulos Marco Antonio Sobrevilla Cabezudo Arash Eshghi Ioannis Papaioannou Verena Rieser Ioannis Konstas
Large language models are known to produce output which sounds fluent and
convincing, but is also often wrong, e.g. "unfaithful" with respect to a
rationale as retrieved from a knowledge base. In this paper, we show that
task-based systems which exhibit certain advanced linguistic dialog behaviors,
such as lexical alignment (repeating what the user said), are in fact preferred
and trusted more, whereas other phenomena, such as pronouns and ellipsis are
dis-preferred. We use open-domain question answering systems as our test-bed
for task based dialog generation and compare several open- and closed-book
models. Our results highlight the danger of systems that appear to be
trustworthy by parroting user input while providing an unfaithful response.