Cognitive Dissonance: Why Do Language Model Outputs Disagree with
  Internal Representations of Truthfulness?
Kevin Liu Stephen Casper Dylan Hadfield-Menell Jacob Andreas
Neural language models (LMs) can be used to evaluate the truth of factual
statements in two ways: they can be either queried for statement probabilities,
or probed for internal representations of truthfulness. Past work has found
that these two procedures sometimes disagree, and that probes tend to be more
accurate than LM outputs. This has led some researchers to conclude that LMs
"lie" or otherwise encode non-cooperative communicative intents. Is this an
accurate description of today's LMs, or can query-probe disagreement arise in
other ways? We identify three different classes of disagreement, which we term
confabulation, deception, and heterogeneity. In many cases, the superiority of
probes is simply attributable to better calibration on uncertain answers rather
than a greater fraction of correct, high-confidence answers. In some cases,
queries and probes perform better on different subsets of inputs, and accuracy
can further be improved by ensembling the two. Code is available at
github.com/lingo-mit/lm-truthfulness.