LIEDER: Linguistically-Informed Evaluation for Discourse Entity
  Recognition
Xiaomeng Zhu Robert Frank
Discourse Entity (DE) recognition is the task of identifying novel and known
entities introduced within a text. While previous work has found that large
language models have basic, if imperfect, DE recognition abilities (Schuster
and Linzen, 2022), it remains largely unassessed which of the fundamental
semantic properties that govern the introduction and subsequent reference to
DEs they have knowledge of. We propose the Linguistically-Informed Evaluation
for Discourse Entity Recognition (LIEDER) dataset that allows for a detailed
examination of language models' knowledge of four crucial semantic properties:
existence, uniqueness, plurality, and novelty. We find evidence that
state-of-the-art large language models exhibit sensitivity to all of these
properties except novelty, which demonstrates that they have yet to reach
human-level language understanding abilities.