How do Language Models Bind Entities in Context?
Jiahai Feng Jacob Steinhardt
To correctly use in-context information, language models (LMs) must bind
entities to their attributes. For example, given a context describing a "green
square" and a "blue circle", LMs must bind the shapes to their respective
colors. We analyze LM representations and identify the binding ID mechanism: a
general mechanism for solving the binding problem, which we observe in every
sufficiently large model from the Pythia and LLaMA families. Using causal
interventions, we show that LMs' internal activations represent binding
information by attaching binding ID vectors to corresponding entities and
attributes. We further show that binding ID vectors form a continuous subspace,
in which distances between binding ID vectors reflect their discernability.
Overall, our results uncover interpretable strategies in LMs for representing
symbolic knowledge in-context, providing a step towards understanding general
in-context reasoning in large-scale LMs.