Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison
  Scaling of Texts with Large Language Models
Patrick Y. Wu Jonathan Nagler Joshua A. Tucker Solomon Messing
Existing text scaling methods often require a large corpus, struggle with
short texts, or require labeled data. We develop a text scaling method that
leverages the pattern recognition capabilities of generative large language
models (LLMs). Specifically, we propose concept-guided chain-of-thought
(CGCoT), which uses prompts designed to summarize ideas and identify target
parties in texts to generate concept-specific breakdowns, in many ways similar
to guidance for human coder content analysis. CGCoT effectively shifts pairwise
text comparisons from a reasoning problem to a pattern recognition problem. We
then pairwise compare concept-specific breakdowns using an LLM. We use the
results of these pairwise comparisons to estimate a scale using the
Bradley-Terry model. We use this approach to scale affective speech on Twitter.
Our measures correlate more strongly with human judgments than alternative
approaches like Wordfish. Besides a small set of pilot data to develop the
CGCoT prompts, our measures require no additional labeled data and produce
binary predictions comparable to a RoBERTa-Large model fine-tuned on thousands
of human-labeled tweets. We demonstrate how combining substantive knowledge
with LLMs can create state-of-the-art measures of abstract concepts.