Robust Pronoun Use Fidelity with English LLMs: Are they Reasoning,
  Repeating, or Just Biased?
Vagrant Gautam Eileen Bingert Dawei Zhu Anne Lauscher Dietrich Klakow
Robust, faithful and harm-free pronoun use for individuals is an important
goal for language models as their use increases, but prior work tends to study
only one or two of these components at a time. To measure progress towards the
combined goal, we introduce the task of pronoun use fidelity: given a context
introducing a co-referring entity and pronoun, the task is to reuse the correct
pronoun later, independent of potential distractors. We present a
carefully-designed dataset of over 5 million instances to evaluate pronoun use
fidelity in English, and we use it to evaluate 37 popular large language models
across architectures (encoder-only, decoder-only and encoder-decoder) and
scales (11M-70B parameters). We find that while models can mostly faithfully
reuse previously-specified pronouns in the presence of no distractors, they are
significantly worse at processing she/her/her, singular they and neopronouns.
Additionally, models are not robustly faithful to pronouns, as they are easily
distracted. With even one additional sentence containing a distractor pronoun,
accuracy drops on average by 34%. With 5 distractor sentences, accuracy drops
by 52% for decoder-only models and 13% for encoder-only models. We show that
widely-used large language models are still brittle, with large gaps in
reasoning and in processing different pronouns in a setting that is very simple
for humans, and we encourage researchers in bias and reasoning to bridge them.