Who Are All The Stochastic Parrots Imitating? They Should Tell Us!
Sagi Shaier Lawrence E. Hunter Katharina von der Wense
Both standalone language models (LMs) as well as LMs within downstream-task
systems have been shown to generate statements which are factually untrue. This
problem is especially severe for low-resource languages, where training data is
scarce and of worse quality than for high-resource languages. In this opinion
piece, we argue that LMs in their current state will never be fully trustworthy
in critical settings and suggest a possible novel strategy to handle this
issue: by building LMs such that can cite their sources - i.e., point a user to
the parts of their training data that back up their outputs. We first discuss
which current NLP tasks would or would not benefit from such models. We then
highlight the expected benefits such models would bring, e.g., quick
verifiability of statements. We end by outlining the individual tasks that
would need to be solved on the way to developing LMs with the ability to cite.
We hope to start a discussion about the field's current approach to building
LMs, especially for low-resource languages, and the role of the training data
in explaining model generations.