Assessing Generalization for Subpopulation Representative Modeling via
  In-Context Learning
Gabriel Simmons Vladislav Savinov
This study evaluates the ability of Large Language Model (LLM)-based
Subpopulation Representative Models (SRMs) to generalize from empirical data,
utilizing in-context learning with data from the 2016 and 2020 American
National Election Studies. We explore generalization across response variables
and demographic subgroups. While conditioning with empirical data improves
performance on the whole, the benefit of in-context learning varies
considerably across demographics, sometimes hurting performance for one
demographic while helping performance for others. The inequitable benefits of
in-context learning for SRM present a challenge for practitioners implementing
SRMs, and for decision-makers who might come to rely on them. Our work
highlights a need for fine-grained benchmarks captured from diverse
subpopulations that test not only fidelity but generalization.