Do Androids Know They're Only Dreaming of Electric Sheep?
Sky CH-Wang Benjamin Van Durme Jason Eisner Chris Kedzie
We design probes trained on the internal representations of a transformer
language model that are predictive of its hallucinatory behavior on in-context
generation tasks. To facilitate this detection, we create a span-annotated
dataset of organic and synthetic hallucinations over several tasks. We find
that probes trained on the force-decoded states of synthetic hallucinations are
generally ecologically invalid in organic hallucination detection. Furthermore,
hidden state information about hallucination appears to be task and
distribution-dependent. Intrinsic and extrinsic hallucination saliency varies
across layers, hidden state types, and tasks; notably, extrinsic hallucinations
tend to be more salient in a transformer's internal representations.
Outperforming multiple contemporary baselines, we show that probing is a
feasible and efficient alternative to language model hallucination evaluation
when model states are available.