Context versus Prior Knowledge in Language Models
Kevin Du Vésteinn Snæbjarnarson Niklas Stoehr Jennifer C. White Aaron Schein Ryan Cotterell
To answer a question, language models often need to integrate prior knowledge
learned during pretraining and new information presented in context. We
hypothesize that models perform this integration in a predictable way across
different questions and contexts: models will rely more on prior knowledge for
questions about entities (e.g., persons, places, etc.) that they are more
familiar with due to higher exposure in the training corpus, and be more easily
persuaded by some contexts than others. To formalize this problem, we propose
two mutual information-based metrics to measure a model's dependency on a
context and on its prior about an entity: first, the persuasion score of a
given context represents how much a model depends on the context in its
decision, and second, the susceptibility score of a given entity represents how
much the model can be swayed away from its original answer distribution about
an entity. Following well-established measurement modeling methods, we
empirically test for the validity and reliability of these metrics. Finally, we
explore and find a relationship between the scores and the model's expected
familiarity with an entity, and provide two use cases to illustrate their
benefits.