Knibbe. 2020. Adhering, Steering, and Queering:
Treatment of Gender in Natural Language Generation. In Proceedings of the Conference on Human
Factors in Computing Systems (CHI) , Honolulu, HI.Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang,
Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth
Belding, Kai-Wei Chang, and William Yang Wang.
2019. Mitigating Gender Bias in Natural Language Processing: Literature Review. In Proceedings of the Association for Computational Linguistics (ACL) , pages 1630–1640, Florence, Italy.
Adam Sutton, Thomas Lansdall-Welfare, and Nello
Cristianini. 2018. Biased embeddings from wild
data: Measuring, understanding and removing.
InProceedings of the International Symposium
on Intelligent Data Analysis , pages 328–339, ’sHertogenbosch, Netherlands.
Chris Sweeney and Maryam Najaﬁan. 2019. A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings. In Proceedings of the Association for Computational Linguistics (ACL) , pages 1662–1667, Florence, Italy.
Chris Sweeney and Maryam Najaﬁan. 2020. Reducing sentiment polarity for demographic attributes


Chris Sweeney and Maryam Najaﬁan. 2020. Reducing sentiment polarity for demographic attributes
in word embeddings using adversarial learning. In
Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 359–368,
Barcelona, Spain.
Nathaniel Swinger, Maria De-Arteaga, Neil Thomas
Heffernan, Mark D.M. Leiserson, and Adam Tauman Kalai. 2019. What are the biases in my word
embedding? In Proceedings of the Conference on
Artiﬁcial Intelligence, Ethics, and Society (AIES) ,
Honolulu, HI.
Samson Tan, Shaﬁq Joty, Min-Yen Kan, and Richard
Socher. 2020. It’s Morphin’ Time! Combating
Linguistic Discrimination with Inﬂectional Perturbations. In Proceedings of the Association for Computational Linguistics (ACL) .
Yi Chern Tan and L. Elisa Celis. 2019. Assessing
Social and Intersectional Biases in Contextualized
Word Representations. In Proceedings of the Conference on Neural Information Processing Systems ,
Vancouver, Canada.
J. Michael Terry, Randall Hendrick, Evangelos Evangelou, and Richard L. Smith. 2010. Variable

Language (Technology) is Power: A Critical Survey of "Bias" in NLP

Summary:
We survey 146 papers analyzing "bias" in NLP systems, finding that their
motivations are often vague, inconsistent, and lacking in normative reasoning,
despite the fact that analyzing "bias" is an inherently normative process. We
further find that these papers' proposed quantitative techniques for measuring
or mitigating "bias" are poorly matched to their motivations and do not engage
with the relevant literature outside of NLP. Based on these findings, we
describe the beginnings of a path forward by proposing three recommendations
that should guide work analyzing "bias" in NLP systems. These recommendations
rest on a greater recognition of the relationships between language and social
hierarchies, encouraging researchers and practitioners to articulate their
conceptualizations of "bias"---i.e., what kinds of system behaviors are
harmful, in what ways, to whom, and why, as well as the normative reasoning
underlying these statements---and to center work around the lived experiences
of members of communities affected by NLP systems, while interrogating and
reimagining the power relations between technologists and such communities.

Su Lin Blodgett
Solon Barocas
Hal Daumé III
Hanna Wallach


Vancouver, Canada.
J. Michael Terry, Randall Hendrick, Evangelos Evangelou, and Richard L. Smith. 2010. Variable
dialect switching among African American children: Inferences about working memory. Lingua ,
120(10):2463–2475.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill.
2013. A Report on the First Native Language Identiﬁcation Shared Task. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications , pages 48–57, Atlanta, GA.
Mike Thelwall. 2018. Gender Bias in Sentiment Analysis. Online Information Review , 42(1):45–57.
Kristen Vaccaro, Karrie Karahalios, Deirdre K. Mulligan, Daniel Kluttz, and Tad Hirsch. 2019. Contestability in Algorithmic Systems. In Conference
Companion Publication of the 2019 on Computer
Supported Cooperative Work and Social Computing ,
pages 523–527, Austin, TX.
Ameya Vaidya, Feng Mai, and Yue Ning. 2019. Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection. arXiv
preprint arXiv:1909.09758v2 .
Eva Vanmassenhove, Christian Hardmeier, and Andy


preprint arXiv:1909.09758v2 .
Eva Vanmassenhove, Christian Hardmeier, and Andy
Way. 2018. Getting Gender Right in Neural Machine Translation. In Proceedings of Empirical
Methods in Natural Language Processing (EMNLP) ,
pages 3003–3008, Brussels, Belgium.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov,
Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal Mediation Analysis for
Interpreting Neural NLP: The Case of Gender Bias.
arXiv preprint arXiv:2004.12265 .
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caiming Xiong. 2020. Double-Hard Debias: Tailoring
Word Embeddings for Gender Bias Mitigation. In
Proceedings of the Association for Computational
Linguistics (ACL) .
Zili Wang. 2019. MSnet: A BERT-based Network for
Gendered Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 89–95, Florence, Italy.


Gendered Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 89–95, Florence, Italy.
Kellie Webster, Marta R. Costa-jussà, Christian Hardmeier, and Will Radford. 2019. Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias
in NLP Workshop 2019. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 1–7, Florence, Italy.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced
corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–618.
Walt Wolfram and Natalie Schilling. 2015. American
English: Dialects and Variation , 3 edition. Wiley
Blackwell.
Austin P. Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Diyi
Yang, and Duen Horng (Polo) Chau. 2020. RECAST: Interactive Auditing of Automatic Toxicity
Detection Models. In Proceedings of the Conference on Human Factors in Computing Systems
(CHI) , Honolulu, HI.
Yinchuan Xu and Junlin Yang. 2019. Look again at


Detection Models. In Proceedings of the Conference on Human Factors in Computing Systems
(CHI) , Honolulu, HI.
Yinchuan Xu and Junlin Yang. 2019. Look again at
the syntax: Relational graph convolutional network
for gendered ambiguous pronoun resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 96–101, Florence,
Italy.Kai-Chou Yang, Timothy Niven, Tzu-Hsuan Chou, and
Hung-Yu Kao. 2019. Fill the GAP: Exploiting
BERT for Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 102–106, Florence, Italy.
Zekun Yang and Juan Feng. 2020. A Causal Inference
Method for Reducing Gender Bias in Word Embedding Relations. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence (AAAI) , New York,
NY .
Daisy Yoo, Anya Ernest, Soﬁa Serholt, Eva Eriksson,
and Peter Dalsgaard. 2019. Service Design in HCI
Research: The Extended Value Co-creation Model.
InProceedings of the Halfway to the Future Symposium , Nottingham, United Kingdom.
Brian Hu Zhang, Blake Lemoine, and Margaret


InProceedings of the Halfway to the Future Symposium , Nottingham, United Kingdom.
Brian Hu Zhang, Blake Lemoine, and Margaret
Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the Conference
on Artiﬁcial Intelligence, Ethics, and Society (AIES) ,
New Orleans, LA.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020a. Demographics
Should Not Be the Reason of Toxicity: Mitigating
Discrimination in Text Classiﬁcations with Instance
Weighting. In Proceedings of the Association for
Computational Linguistics (ACL) .
Haoran Zhang, Amy X. Lu, Mohamed Abdalla,
Matthew McDermott, and Marzyeh Ghassemi.
2020b. Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings. In Proceedings
of the ACM Conference on Health, Inference, and
Learning .
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini,
Kai-Wei Chang, and Ahmed Hassan Awadallah.
2020. Gender Bias in Multilingual Embeddings and
Cross-Lingual Transfer. In Proceedings of the Association for Computational Linguistics (ACL) .


2020. Gender Bias in Multilingual Embeddings and
Cross-Lingual Transfer. In Proceedings of the Association for Computational Linguistics (ACL) .
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019.
Gender Bias in Contextualized Word Embeddings.
InProceedings of the North American Association
for Computational Linguistics (NAACL) , pages 629–
634, Minneapolis, MN.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like
shopping: Reducing gender bias ampliﬁcation using corpus-level constraints. In Proceedings of
Empirical Methods in Natural Language Processing (EMNLP) , pages 2979–2989, Copenhagen, Denmark.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender Bias
in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the North American
Association for Computational Linguistics (NAACL) ,
pages 15–20, New Orleans, LA.


Association for Computational Linguistics (NAACL) ,
pages 15–20, New Orleans, LA.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and KaiWei Chang. 2018b. Learning Gender-Neutral Word
Embeddings. In Proceedings of Empirical Methods
in Natural Language Processing (EMNLP) , pages
4847–4853, Brussels, Belgium.
Alina Zhiltsova, Simon Caton, and Catherine Mulwa.
2019. Mitigation of Unintended Biases against NonNative English Texts in Sentiment Analysis. In Proceedings of the Irish Conference on Artiﬁcial Intelligence and Cognitive Science , Galway, Ireland.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang,
Muhao Chen, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical genders. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP) , pages 5279–
5287, Hong Kong, China.
Ran Zmigrod, S. J. Mielke, Hanna Wallach, and Ryan
Cotterell. 2019. Counterfactual data augmentation
for mitigating gender stereotypes in languages with
rich morphology. In Proceedings of the Association


Cotterell. 2019. Counterfactual data augmentation
for mitigating gender stereotypes in languages with
rich morphology. In Proceedings of the Association
for Computational Linguistics (ACL) , pages 1651–
1661, Florence, Italy.
A Appendix
In Table 3, we provide examples of the papers’ motivations and techniques across several NLP tasks.
A.1 Categorization details
In this section, we provide some additional details
about our method—speciﬁcally, our categorization.
What counts as being covered by an NLP task?
We considered a paper to cover a given NLP task if
it analyzed “bias” with respect to that task, but not
if it only evaluated overall performance on that task.
For example, a paper examining the impact of mitigating “bias” in word embeddings on “bias” in sentiment analysis would be counted as covering both
NLP tasks. In contrast, a paper assessing whether
performance on sentiment analysis degraded after
mitigating “bias” in word embeddings would be
counted only as focusing on embeddings.
What counts as a motivation? We considered a
motivation to include any description of the problem that motivated the paper or proposed quantitative technique, including any normative reasoning.


What counts as a motivation? We considered a
motivation to include any description of the problem that motivated the paper or proposed quantitative technique, including any normative reasoning.
We excluded from the “Vague/unstated” category of motivations the papers that participated in
the Gendered Ambiguous Pronoun (GAP) Shared
Task at the First ACL Workshop on Gender Bias in
NLP. In an ideal world, shared task papers would
engage with “bias” more critically, but given the
nature of shared tasks it is understandable that theydo not. As a result, we excluded them from our
counts for techniques as well. We cite the papers
here; most propose techniques we would have categorized as “Questionable correlations,” with a few
as “Other representational harms” (Abzaliev, 2019;
Attree, 2019; Bao and Qiao, 2019; Chada, 2019;
Ionita et al., 2019; Liu, 2019; Lois et al., 2019;
Wang, 2019; Xu and Yang, 2019; Yang et al., 2019).
We excluded Dabas et al. (2020) from our survey
because we could not determine what this paper’s
user study on fairness was actually measuring.
Finally, we actually categorized the motivation


because we could not determine what this paper’s
user study on fairness was actually measuring.
Finally, we actually categorized the motivation
for Liu et al. (2019) (i.e., the last row in Table 3) as
“Questionable correlations” due to a sentence elsewhere in the paper; had the paragraph we quoted
been presented without more detail, we would have
categorized the motivation as “Vague/unstated.”
A.2 Full categorization: Motivations
Allocational harms Hovy and Spruit (2016);
Caliskan et al. (2017); Madnani et al. (2017);
Dixon et al. (2018); Kiritchenko and Mohammad
(2018); Shen et al. (2018); Zhao et al. (2018b);
Bhaskaran and Bhallamudi (2019); Bordia and
Bowman (2019); Brunet et al. (2019); Chaloner
and Maldonado (2019); De-Arteaga et al. (2019);
Dev and Phillips (2019); Font and Costa-jussà
(2019); James-Sorenson and Alvarez-Melis (2019);
Kurita et al. (2019); Mayﬁeld et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane


Kurita et al. (2019); Mayﬁeld et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane
et al. (2019); Sedoc and Ungar (2019); Sun et al.
(2019); Zmigrod et al. (2019); Hutchinson et al.
(2020); Papakyriakopoulos et al. (2020); Ravfogel et al. (2020); Strengers et al. (2020); Sweeney
and Najaﬁan (2020); Tan et al. (2020); Zhang et al.
(2020b).
Stereotyping Bolukbasi et al. (2016a,b);
Caliskan et al. (2017); McCurdy and Serbetçi
(2017); Rudinger et al. (2017); Zhao et al. (2017);
Curry and Rieser (2018); Díaz et al. (2018);
Santana et al. (2018); Sutton et al. (2018); Zhao
et al. (2018a,b); Agarwal et al. (2019); Basta et al.
(2019); Bhaskaran and Bhallamudi (2019); Bordia
and Bowman (2019); Brunet et al. (2019); Cao
and Daumé (2019); Chaloner and Maldonado


and Bowman (2019); Brunet et al. (2019); Cao
and Daumé (2019); Chaloner and Maldonado
(2019); Cho et al. (2019); Dev and Phillips (2019);
Font and Costa-jussà (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);
Kaneko and Bollegala (2019); Karve et al. (2019);
Kurita et al. (2019); Lauscher and Glavaš (2019);
Lee et al. (2019); Manzini et al. (2019); Mayﬁeld
Categories
NLP task Stated motivation Motivations Techniques
Language
modeling
(Bordia and
Bowman,
2019)“Existing biases in data can be ampliﬁed by models and the
resulting output consumed by the public can inﬂuence them ,encourage and reinforce harmful stereotypes , ordistort the truth .
Automated systems that depend on these models can take problematic actions based on biased proﬁling of individuals. ”Allocational
harms,
stereotypingQuestionable
correlations
Sentiment
analysis
(Kiritchenko
and
Mohammad,


harms,
stereotypingQuestionable
correlations
Sentiment
analysis
(Kiritchenko
and
Mohammad,
2018)“Other biases can be inappropriate and result in negative experiences for some groups of people. Examples include, loan
eligibility and crime recidivism prediction systems ...and resumé
sorting systems that believe that men are more qualiﬁed to be
programmers than women (Bolukbasi et al., 2016). Similarly,
sentiment and emotion analysis systems can also perpetuate and
accentuate inappropriate human biases, e.g., systems that consider
utterances from one race or gender to be less positive simply because of their race or gender, or customer support systems that
prioritize a call from an angry male over a call from the equally
angry female . ”Allocational
harms, other
representational
harms (system
performance
differences w.r.t.
text written by
different social
groups)Questionable
correlations
(differences in
sentiment
intensity scores
w.r.t. text about
different social
groups)
Machine
translation
(Cho et al.,
2019)“[MT training] may incur an association of gender-speciﬁed pronouns (in the target) and gender-neutral ones (in the source) for


2019)“[MT training] may incur an association of gender-speciﬁed pronouns (in the target) and gender-neutral ones (in the source) for
lexicon pairs that frequently collocate in the corpora. We claim
that this kind of phenomenon seriously threatens the fairness of a
translation system, in the sense that it lacks generality andinserts
social bias to the inference. Moreover, the input is not fully correct(considering gender-neutrality) and might offend the users
who expect fairer representations. ”Questionable
correlations,
other
representational
harmsQuestionable
correlations
Machine
translation
(Stanovsky
et al., 2019)“Learned models exhibit social bias when their training data
encode stereotypes not relevant for the task, but the correlations
are picked up anyway . ”Stereotyping,
questionable
correlationsStereotyping,
other
representational
harms (system
performance
differences),
questionable
correlations
Type-level
embeddings
(Zhao et al.,
2018b)“However, embeddings trained on human-generated corpora have
been demonstrated to inherit strong gender stereotypes that reﬂect social constructs....Such a bias substantially affects downstream applications ....This concerns the practitioners who use
the embedding model to build gender-sensitive applications such


the embedding model to build gender-sensitive applications such
as a resume ﬁltering system or a job recommendation system as
the automated system may discriminate candidates based on their
gender, as reﬂected by their name. Besides, biased embeddings
may implicitly affect downstream applications used in our daily
lives. For example, when searching for ‘computer scientist’ using
a search engine...a search algorithm using an embedding model in
the backbone tends to rank male scientists higher than females’
[sic], hindering women from being recognized and further exacerbating the gender inequality in the community. ”Allocational
harms,
stereotyping,
other
representational
harmsStereotyping
Type-level
and contextualized
embeddings
(May et al.,
2019)“[P]rominent word embeddings such as word2vec (Mikolov et
al., 2013) and GloVe (Pennington et al., 2014) encode systematic
biases against women and black people (Bolukbasi et al., 2016;
Garg et al., 2018), implicating many NLP systems in scaling up
social injustice. ”Vague Stereotyping
Dialogue
generation
(Liu et al.,
2019)“Since the goal of dialogue systems is to talk with users...if the


Dialogue
generation
(Liu et al.,
2019)“Since the goal of dialogue systems is to talk with users...if the
systems show discriminatory behaviors in the interactions, the
user experience will be adversely affected. Moreover, public commercial chatbots can get resisted for their improper speech. ”Vague/unstated Stereotyping,
other
representational
harms,
questionable
correlations
Table 3: Examples of the categories into which the papers’ motivations and proposed quantitative techniques for
measuring or mitigating “bias” fall. Bold text in the quotes denotes the content that yields our categorizations.
et al. (2019); Précenth (2019); Pujari et al. (2019);
Ruane et al. (2019); Stanovsky et al. (2019);
Sun et al. (2019); Tan and Celis (2019); Webster
et al. (2019); Zmigrod et al. (2019); Gyamﬁ et al.
(2020); Hube et al. (2020); Hutchinson et al.
(2020); Kim et al. (2020); Nadeem et al. (2020);
Papakyriakopoulos et al. (2020); Ravfogel et al.
(2020); Rozado (2020); Sen and Ganguly (2020);


Papakyriakopoulos et al. (2020); Ravfogel et al.
(2020); Rozado (2020); Sen and Ganguly (2020);
Shin et al. (2020); Strengers et al. (2020).
Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi
et al. (2016b); Hovy and Spruit (2016); Blodgett
and O’Connor (2017); Larson (2017); Schnoebelen
(2017); Blodgett et al. (2018); Curry and Rieser
(2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Park et al. (2018);
Shen et al. (2018); Thelwall (2018); Zhao et al.
(2018b); Badjatiya et al. (2019); Bagdasaryan et al.
(2019); Bamman et al. (2019); Cao and Daumé
(2019); Chaloner and Maldonado (2019); Cho et al.
(2019); Davidson et al. (2019); De-Arteaga et al.
(2019); Fisher (2019); Font and Costa-jussà (2019);


(2019); Davidson et al. (2019); De-Arteaga et al.
(2019); Fisher (2019); Font and Costa-jussà (2019);
Garimella et al. (2019); Loukina et al. (2019); Mayﬁeld et al. (2019); Mehrabi et al. (2019); Nozza
et al. (2019); Prabhakaran et al. (2019); Romanov
et al. (2019); Ruane et al. (2019); Sap et al. (2019);
Sheng et al. (2019); Sun et al. (2019); Sweeney
and Najaﬁan (2019); Vaidya et al. (2019); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020);
Hutchinson et al. (2020); Kim et al. (2020); Peng
et al. (2020); Rios (2020); Sap et al. (2020); Shah
et al. (2020); Sheng et al. (2020); Tan et al. (2020);
Zhang et al. (2020a,b).
Questionable correlations Jørgensen et al.
(2015); Hovy and Spruit (2016); Madnani et al.
(2017); Rudinger et al. (2017); Zhao et al. (2017);


(2015); Hovy and Spruit (2016); Madnani et al.
(2017); Rudinger et al. (2017); Zhao et al. (2017);
Burns et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Lu et al. (2018);
Park et al. (2018); Shen et al. (2018); Zhang
et al. (2018); Badjatiya et al. (2019); Bhargava
and Forsyth (2019); Cao and Daumé (2019); Cho
et al. (2019); Davidson et al. (2019); Dev et al.
(2019); Garimella et al. (2019); Garg et al. (2019);
Huang et al. (2019); James-Sorenson and AlvarezMelis (2019); Kaneko and Bollegala (2019); Liu
et al. (2019); Karve et al. (2019); Nozza et al.
(2019); Prabhakaran et al. (2019); Romanov et al.
(2019); Sap et al. (2019); Sedoc and Ungar (2019);
Stanovsky et al. (2019); Sweeney and Najaﬁan


(2019); Sap et al. (2019); Sedoc and Ungar (2019);
Stanovsky et al. (2019); Sweeney and Najaﬁan
(2019); Vaidya et al. (2019); Zhiltsova et al. (2019);Chopra et al. (2020); Gonen and Webster (2020);
Gyamﬁ et al. (2020); Hube et al. (2020); Ravfogel
et al. (2020); Rios (2020); Ross et al. (2020); Saunders and Byrne (2020); Sen and Ganguly (2020);
Shah et al. (2020); Sweeney and Najaﬁan (2020);
Yang and Feng (2020); Zhang et al. (2020a).
Vague/unstated Rudinger et al. (2018); Webster
et al. (2018); Dinan et al. (2019); Florez (2019);
Jumelet et al. (2019); Lauscher et al. (2019); Liang
et al. (2019); Maudslay et al. (2019); May et al.
(2019); Prates et al. (2019); Prost et al. (2019);
Qian et al. (2019); Swinger et al. (2019); Zhao


(2019); Prates et al. (2019); Prost et al. (2019);
Qian et al. (2019); Swinger et al. (2019); Zhao
et al. (2019); Zhou et al. (2019); Ethayarajh (2020);
Huang et al. (2020); Jia et al. (2020); Popovi ´c et al.
(2020); Pryzant et al. (2020); Vig et al. (2020);
Wang et al. (2020); Zhao et al. (2020).
Surveys, frameworks, and meta-analyses
Hovy and Spruit (2016); Larson (2017); McCurdy
and Serbetçi (2017); Schnoebelen (2017); Basta
et al. (2019); Ethayarajh et al. (2019); Gonen and
Goldberg (2019); Lauscher and Glavaš (2019);
Loukina et al. (2019); Mayﬁeld et al. (2019);
Mirzaev et al. (2019); Prabhumoye et al. (2019);
Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);


Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);
Shah et al. (2020); Strengers et al. (2020); Wright
et al. (2020).
B Full categorization: Techniques
Allocational harms De-Arteaga et al. (2019);
Prost et al. (2019); Romanov et al. (2019); Zhao
et al. (2020).
Stereotyping Bolukbasi et al. (2016a,b);
Caliskan et al. (2017); McCurdy and Serbetçi
(2017); Díaz et al. (2018); Santana et al. (2018);
Sutton et al. (2018); Zhang et al. (2018); Zhao
et al. (2018a,b); Agarwal et al. (2019); Basta et al.
(2019); Bhaskaran and Bhallamudi (2019); Brunet
et al. (2019); Cao and Daumé (2019); Chaloner
and Maldonado (2019); Dev and Phillips (2019);
Ethayarajh et al. (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);


Ethayarajh et al. (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);
Jumelet et al. (2019); Kaneko and Bollegala
(2019); Karve et al. (2019); Kurita et al. (2019);
Lauscher and Glavaš (2019); Lauscher et al.
(2019); Lee et al. (2019); Liang et al. (2019); Liu
et al. (2019); Manzini et al. (2019); Maudslay et al.
(2019); May et al. (2019); Mirzaev et al. (2019);
Prates et al. (2019); Précenth (2019); Prost et al.
(2019); Pujari et al. (2019); Qian et al. (2019);
Sedoc and Ungar (2019); Stanovsky et al. (2019);
Tan and Celis (2019); Zhao et al. (2019); Zhou
et al. (2019); Chopra et al. (2020); Gyamﬁ et al.
(2020); Nadeem et al. (2020); Nissim et al. (2020);
Papakyriakopoulos et al. (2020); Popovi ´c et al.


Papakyriakopoulos et al. (2020); Popovi ´c et al.
(2020); Ravfogel et al. (2020); Ross et al. (2020);
Rozado (2020); Saunders and Byrne (2020); Shin
et al. (2020); Vig et al. (2020); Wang et al. (2020);
Yang and Feng (2020); Zhao et al. (2020).
Other representational harms Jørgensen et al.
(2015); Hovy and Søgaard (2015); Blodgett et al.
(2016); Blodgett and O’Connor (2017); Blodgett
et al. (2018); Curry and Rieser (2018); Dixon et al.
(2018); Park et al. (2018); Thelwall (2018); Webster et al. (2018); Badjatiya et al. (2019); Bagdasaryan et al. (2019); Bamman et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019);
Font and Costa-jussà (2019); Garg et al. (2019);
Garimella et al. (2019); Liu et al. (2019); Loukina et al. (2019); Mehrabi et al. (2019); Nozza


Garimella et al. (2019); Liu et al. (2019); Loukina et al. (2019); Mehrabi et al. (2019); Nozza
et al. (2019); Sap et al. (2019); Sheng et al. (2019);
Stanovsky et al. (2019); Vaidya et al. (2019);
Webster et al. (2019); Ethayarajh (2020); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020);
Huang et al. (2020); Kim et al. (2020); Peng et al.
(2020); Ravfogel et al. (2020); Rios (2020); Sap
et al. (2020); Saunders and Byrne (2020); Sheng
et al. (2020); Sweeney and Najaﬁan (2020); Tan
et al. (2020); Zhang et al. (2020a,b).
Questionable correlations Jurgens et al. (2017);
Madnani et al. (2017); Rudinger et al. (2017);
Zhao et al. (2017); Burns et al. (2018); Díaz
et al. (2018); Kiritchenko and Mohammad (2018);
Lu et al. (2018); Rudinger et al. (2018); Shen


et al. (2018); Kiritchenko and Mohammad (2018);
Lu et al. (2018); Rudinger et al. (2018); Shen
et al. (2018); Bordia and Bowman (2019); Cao
and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Dinan et al.
(2019); Fisher (2019); Florez (2019); Font and
Costa-jussà (2019); Garg et al. (2019); Huang et al.
(2019); Liu et al. (2019); Nozza et al. (2019);
Prabhakaran et al. (2019); Qian et al. (2019); Sap
et al. (2019); Stanovsky et al. (2019); Sweeney and
Najaﬁan (2019); Swinger et al. (2019); Zhiltsova
et al. (2019); Zmigrod et al. (2019); Hube et al.
(2020); Hutchinson et al. (2020); Jia et al. (2020);
Papakyriakopoulos et al. (2020); Popovi ´c et al.
(2020); Pryzant et al. (2020); Saunders and Byrne
(2020); Sen and Ganguly (2020); Shah et al. (2020);


(2020); Pryzant et al. (2020); Saunders and Byrne
(2020); Sen and Ganguly (2020); Shah et al. (2020);
Sweeney and Najaﬁan (2020); Zhang et al. (2020b).
Vague/unstated None.Surveys, frameworks, and meta-analyses
Hovy and Spruit (2016); Larson (2017); McCurdy
and Serbetçi (2017); Schnoebelen (2017); Basta
et al. (2019); Ethayarajh et al. (2019); Gonen and
Goldberg (2019); Lauscher and Glavaš (2019);
Loukina et al. (2019); Mayﬁeld et al. (2019);
Mirzaev et al. (2019); Prabhumoye et al. (2019);
Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);
Shah et al. (2020); Strengers et al. (2020); Wright
et al. (2020).


Language Models are Few-Shot Learners
Tom B. BrownBenjamin MannNick RyderMelanie Subbiah
Jared KaplanyPrafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry
Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan
Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter
Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray
Benjamin Chess Jack Clark Christopher Berner
Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei
OpenAI
Abstract
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training
on a large corpus of text followed by ﬁne-tuning on a speciﬁc task. While typically task-agnostic
in architecture, this method still requires task-speciﬁc ﬁne-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language task from only
a few examples or from simple instructions – something which current NLP systems still largely
struggle to do. Here we show that scaling up language models greatly improves task-agnostic,

Language Models are Few-Shot Learners

Summary:
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - something which
current NLP systems still largely struggle to do. Here we show that scaling up
language models greatly improves task-agnostic, few-shot performance, sometimes
even reaching competitiveness with prior state-of-the-art fine-tuning
approaches. Specifically, we train GPT-3, an autoregressive language model with
175 billion parameters, 10x more than any previous non-sparse language model,
and test its performance in the few-shot setting. For all tasks, GPT-3 is
applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require
on-the-fly reasoning or domain adaptation, such as unscrambling words, using a
novel word in a sentence, or performing 3-digit arithmetic. At the same time,
we also identify some datasets where GPT-3's few-shot learning still struggles,
as well as some datasets where GPT-3 faces methodological issues related to
training on large web corpora. Finally, we find that GPT-3 can generate samples
of news articles which human evaluators have difficulty distinguishing from
articles written by humans. We discuss broader societal impacts of this finding
and of GPT-3 in general.

Tom B. Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell
Sandhini Agarwal
Ariel Herbert-Voss
Gretchen Krueger
Tom Henighan
Rewon Child
Aditya Ramesh
Daniel M. Ziegler
Jeffrey Wu
Clemens Winter
Christopher Hesse
Mark Chen
Eric Sigler
Mateusz Litwin
Scott Gray
Benjamin Chess
Jack Clark
Christopher Berner
Sam McCandlish
Alec Radford
Ilya Sutskever
Dario Amodei

40+32 pages

Learning to learn by gradient descent by gradient descent

Massively multilingual neural machine translation

Roee Aharoni, Melvin Johnson, Orhan Firat

Language (technology) is power: A critical survey of “bias” in NLP


a few examples or from simple instructions – something which current NLP systems still largely
struggle to do. Here we show that scaling up language models greatly improves task-agnostic,
few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art ﬁnetuning approaches. Speciﬁcally, we train GPT-3, an autoregressive language model with 175 billion
parameters, 10x more than any previous non-sparse language model, and test its performance in
the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or ﬁne-tuning,
with tasks and few-shot demonstrations speciﬁed purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation, question-answering, and
cloze tasks, as well as several tasks that require on-the-ﬂy reasoning or domain adaptation, such as
unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same
time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some
datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally,

Language Models are Few-Shot Learners

40+32 pages


datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally,
we ﬁnd that GPT-3 can generate samples of news articles which human evaluators have difﬁculty
distinguishing from articles written by humans. We discuss broader societal impacts of this ﬁnding
and of GPT-3 in general.
Equal contribution
yJohns Hopkins University, OpenAI
Author contributions listed at end of paper.arXiv:2005.14165v4  [cs.CL]  22 Jul 2020
Contents
1 Introduction 3
2 Approach 6
2.1 Model and Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Training Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Language Models are Few-Shot Learners

40+32 pages


Language Models are Few-Shot Learners

40+32 pages


4 Measuring and Preventing Memorization Of Benchmarks 29
5 Limitations 33
6 Broader Impacts 34
6.1 Misuse of Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2 Fairness, Bias, and Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.3 Energy Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
7 Related Work 39
8 Conclusion 40
A Details of Common Crawl Filtering 43
B Details of Model Training 43
C Details of Test Set Contamination Studies 43
D Total Compute Used to Train Language Models 46
E Human Quality Assessment of Synthetic News Articles 46
F Additional Samples from GPT-3 48
G Details of Task Phrasing and Speciﬁcations 50
H Results on All Tasks for All Model Sizes 63
2
1 Introduction

Language Models are Few-Shot Learners

40+32 pages


G Details of Task Phrasing and Speciﬁcations 50
H Results on All Tasks for All Model Sizes 63
2
1 Introduction
Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly
ﬂexible and task-agnostic ways for downstream transfer. First, single-layer representations were learned using word
vectors [ MCCD13 ,PSM14 ] and fed to task-speciﬁc architectures, then RNNs with multiple layers of representations
and contextual state were used to form stronger representations [ DL15 ,MBXS17 ,PNZtY18 ] (though still applied to
task-speciﬁc architectures), and more recently pre-trained recurrent or transformer language models [ VSP+17] have
been directly ﬁne-tuned, entirely removing the need for task-speciﬁc architectures [RNSS18, DCLT18, HR18].
This last paradigm has led to substantial progress on many challenging NLP tasks such as reading comprehension,
question answering, textual entailment, and many others, and has continued to advance based on new architectures
and algorithms [ RSR+19,LOG+19,YDY+19,LCG+19]. However, a major limitation to this approach is that while

Language Models are Few-Shot Learners

40+32 pages


and algorithms [ RSR+19,LOG+19,YDY+19,LCG+19]. However, a major limitation to this approach is that while
the architecture is task-agnostic, there is still a need for task-speciﬁc datasets and task-speciﬁc ﬁne-tuning: to achieve
strong performance on a desired task typically requires ﬁne-tuning on a dataset of thousands to hundreds of thousands
of examples speciﬁc to that task. Removing this limitation would be desirable, for several reasons.
First, from a practical perspective, the need for a large dataset of labeled examples for every new task limits the
applicability of language models. There exists a very wide range of possible useful language tasks, encompassing
anything from correcting grammar, to generating examples of an abstract concept, to critiquing a short story. For many
of these tasks it is difﬁcult to collect a large supervised training dataset, especially when the process must be repeated
for every new task.
Second, the potential to exploit spurious correlations in training data fundamentally grows with the expressiveness
of the model and the narrowness of the training distribution. This can create problems for the pre-training plus
ﬁne-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then

Language Models are Few-Shot Learners

40+32 pages


ﬁne-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then
ﬁne-tuned on very narrow task distributions. For instance [ HLW+20] observe that larger models do not necessarily
generalize better out-of-distribution. There is evidence that suggests that the generalization achieved under this paradigm
can be poor because the model is overly speciﬁc to the training distribution and does not generalize well outside it
[YdC+19,MPL19 ]. Thus, the performance of ﬁne-tuned models on speciﬁc benchmarks, even when it is nominally at
human-level, may exaggerate actual performance on the underlying task [GSL+18, NK19].
Third, humans do not require large supervised datasets to learn most language tasks – a brief directive in natural
language (e.g. “please tell me if this sentence describes something happy or something sad”) or at most a tiny number
of demonstrations (e.g. “here are two examples of people acting brave; please give a third example of bravery”) is often
Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad

Language Models are Few-Shot Learners

40+32 pages


Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad
set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize
the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within
the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a
model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded
within a single sequence.
3
Figure 1.2: Larger models make increasingly efﬁcient use of in-context information. We show in-context learning
performance on a simple task requiring the model to remove random symbols from a word, both with and without a
natural language task description (see Sec. 3.9.2). The steeper “in-context learning curves” for large models demonstrate
improved ability to learn a task from contextual information. We see qualitatively similar behavior across a wide range
of tasks.
sufﬁcient to enable a human to perform a new task to at least a reasonable degree of competence. Aside from pointing
to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans

Language Models are Few-Shot Learners

40+32 pages


to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans
to seamlessly mix together or switch between many tasks and skills, for example performing addition during a lengthy
dialogue. To be broadly useful, we would someday like our NLP systems to have this same ﬂuidity and generality.
One potential route towards addressing these issues is meta-learning1– which in the context of language models means
the model develops a broad set of skills and pattern recognition abilities at training time, and then uses those abilities
at inference time to rapidly adapt to or recognize the desired task (illustrated in Figure 1.1). Recent work [ RWC+19]
attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form
of task speciﬁcation: the model is conditioned on a natural language instruction and/or a few demonstrations of the task
and is then expected to complete further instances of the task simply by predicting what comes next.
While it has shown some initial promise, this approach still achieves results far inferior to ﬁne-tuning – for example
[RWC+19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind

Language Models are Few-Shot Learners

40+32 pages


[RWC+19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind
the state of the art. Meta-learning clearly requires substantial improvement in order to be viable as a practical method of
solving language tasks.
Another recent trend in language modeling may offer a way forward. In recent years the capacity of transformer
language models has increased substantially, from 100 million parameters [ RNSS18 ], to 300 million parameters
[DCLT18 ], to 1.5 billion parameters [ RWC+19], to 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19],
and ﬁnally 17 billion parameters [ Tur20 ]. Each increase has brought improvements in text synthesis and/or downstream
NLP tasks, and there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a
smooth trend of improvement with scale [ KMH+20]. Since in-context learning involves absorbing many skills and
tasks within the parameters of the model, it is plausible that in-context learning abilities might show similarly strong
gains with scale.
1In the context of language models this has sometimes been called “zero-shot transfer”, but this term is potentially ambiguous:
the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time

Language Models are Few-Shot Learners

40+32 pages


the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time
demonstrations to the model, so is not truly learning from zero examples. To avoid this confusion, we use the term “meta-learning”
to capture the inner-loop / outer-loop structure of the general method, and the term “in context-learning” to refer to the inner
loop of meta-learning. We further specialize the description to “zero-shot”, “one-shot”, or “few-shot” depending on how many
demonstrations are provided at inference time. These terms are intended to remain agnostic on the question of whether the model
learns new tasks from scratch at inference time or simply recognizes patterns seen during training – this is an important issue which
we discuss later in the paper, but “meta-learning” is intended to encompass both possibilities, and simply describes the inner-outer
loop structure.
4
Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks While zero-shot performance
improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are
more proﬁcient at in-context learning. See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP
benchmark suite.

Language Models are Few-Shot Learners

40+32 pages


more proﬁcient at in-context learning. See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP
benchmark suite.
In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call
GPT-3, and measuring its in-context learning abilities. Speciﬁcally, we evaluate GPT-3 on over two dozen NLP datasets,
as well as several novel tasks designed to test rapid adaptation to tasks unlikely to be directly contained in the training
set. For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we
allow as many demonstrations as will ﬁt into the model’s context window (typically 10 to 100), (b) “one-shot learning”,
where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only
an instruction in natural language is given to the model. GPT-3 could also in principle be evaluated in the traditional
ﬁne-tuning setting, but we leave this to future work.
Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to

Language Models are Few-Shot Learners

40+32 pages


Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to
remove extraneous symbols from a word. Model performance improves with the addition of a natural language task
description, and with the number of examples in the model’s context, K. Few-shot learning also improves dramatically
with model size. Though the results in this case are particularly striking, the general trends with both model size and
number of examples in-context hold for most tasks we study. We emphasize that these “learning” curves involve no
gradient updates or ﬁne-tuning, just increasing numbers of demonstrations given as conditioning.
Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot
setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held
by ﬁne-tuned models). For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in
the one-shot setting, 85.0 F1 in the few-shot setting. Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the

Language Models are Few-Shot Learners

40+32 pages


the one-shot setting, 85.0 F1 in the few-shot setting. Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the
zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art
relative to ﬁne-tuned models operating in the same closed-book setting.
GPT-3 also displays one-shot and few-shot proﬁciency at tasks designed to test rapid adaption or on-the-ﬂy reasoning,
which include unscrambling words, performing arithmetic, and using novel words in a sentence after seeing them
deﬁned only once. We also show that in the few-shot setting, GPT-3 can generate synthetic news articles which human
evaluators have difﬁculty distinguishing from human-generated articles.
At the same time, we also ﬁnd some tasks on which few-shot performance struggles, even at the scale of GPT-3. This
includes natural language inference tasks like the ANLI dataset, and some reading comprehension datasets like RACE
or QuAC. By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we

Language Models are Few-Shot Learners

40+32 pages


or QuAC. By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we
hope to stimulate study of few-shot learning in language models and draw attention to where progress is most needed.
A heuristic sense of the overall results can be seen in Figure 1.3, which aggregates the various tasks (though it should
not be seen as a rigorous or meaningful benchmark in itself).
5
We also undertake a systematic study of “data contamination” – a growing problem when training high capacity models
on datasets such as Common Crawl, which can potentially include content from test datasets simply because such
content often exists on the web. In this paper we develop systematic tools to measure data contamination and quantify
its distorting effects. Although we ﬁnd that data contamination has a minimal effect on GPT-3’s performance on most
datasets, we do identify a few datasets where it could be inﬂating results, and we either do not report results on these
datasets or we note them with an asterisk, depending on the severity.
In addition to all the above, we also train a series of smaller models (ranging from 125 million parameters to 13 billion
parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings. Broadly, for most

Language Models are Few-Shot Learners

40+32 pages


parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings. Broadly, for most
tasks we ﬁnd relatively smooth scaling with model capacity in all three settings; one notable pattern is that the gap
between zero-, one-, and few-shot performance often grows with model capacity, perhaps suggesting that larger models
are more proﬁcient meta-learners.
Finally, given the broad spectrum of capabilities displayed by GPT-3, we discuss concerns about bias, fairness, and
broader societal impacts, and attempt a preliminary analysis of GPT-3’s characteristics in this regard.
The remainder of this paper is organized as follows. In Section 2, we describe our approach and methods for training
GPT-3 and evaluating it. Section 3 presents results on the full range of tasks in the zero-, one- and few-shot settings.
Section 4 addresses questions of data contamination (train-test overlap). Section 5 discusses limitations of GPT-3.
Section 6 discusses broader impacts. Section 7 reviews related work and Section 8 concludes.
2 Approach
Our basic pre-training approach, including model, data, and training, is similar to the process described in [ RWC+19],
with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training. Our use

Language Models are Few-Shot Learners

40+32 pages


with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training. Our use
of in-context learning is also similar to [ RWC+19], but in this work we systematically explore different settings for
learning within the context. Therefore, we start this section by explicitly deﬁning and contrasting the different settings
that we will be evaluating GPT-3 on or could in principle evaluate GPT-3 on. These settings can be seen as lying on a
spectrum of how much task-speciﬁc data they tend to rely on. Speciﬁcally, we can identify at least four points on this
spectrum (see Figure 2.1 for an illustration):
•Fine-Tuning (FT) has been the most common approach in recent years, and involves updating the weights of
a pre-trained model by training on a supervised dataset speciﬁc to the desired task. Typically thousands to
hundreds of thousands of labeled examples are used. The main advantage of ﬁne-tuning is strong performance
on many benchmarks. The main disadvantages are the need for a new large dataset for every task, the potential
for poor generalization out-of-distribution [ MPL19 ], and the potential to exploit spurious features of the
training data [ GSL+18,NK19 ], potentially resulting in an unfair comparison with human performance. In

Language Models are Few-Shot Learners

40+32 pages


training data [ GSL+18,NK19 ], potentially resulting in an unfair comparison with human performance. In
this work we do not ﬁne-tune GPT-3 because our focus is on task-agnostic performance, but GPT-3 can be
ﬁne-tuned in principle and this is a promising direction for future work.
•Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few
demonstrations of the task at inference time as conditioning [ RWC+19], but no weight updates are allowed.
As shown in Figure 2.1, for a typical dataset an example has a context and a desired completion (for example
an English sentence and the French translation), and few-shot works by giving Kexamples of context and
completion, and then one ﬁnal example of context, with the model expected to provide the completion. We
typically set Kin the range of 10 to 100 as this is how many examples can ﬁt in the model’s context window
(nctx= 2048 ). The main advantages of few-shot are a major reduction in the need for task-speciﬁc data and
reduced potential to learn an overly narrow distribution from a large but narrow ﬁne-tuning dataset. The main

Language Models are Few-Shot Learners

40+32 pages


reduced potential to learn an overly narrow distribution from a large but narrow ﬁne-tuning dataset. The main
disadvantage is that results from this method have so far been much worse than state-of-the-art ﬁne-tuned
models. Also, a small amount of task speciﬁc data is still required. As indicated by the name, few-shot
learning as described here for language models is related to few-shot learning as used in other contexts in
ML [ HYC01 ,VBL+16] – both involve learning based on a broad distribution of tasks (in this case implicit in
the pre-training data) and then rapidly adapting to a new task.
•One-Shot (1S) is the same as few-shot except that only one demonstration is allowed, in addition to a natural
language description of the task, as shown in Figure 1. The reason to distinguish one-shot from few-shot and
zero-shot (below) is that it most closely matches the way in which some tasks are communicated to humans.
For example, when asking humans to generate a dataset on a human worker service (for example Mechanical
Turk), it is common to give one demonstration of the task. By contrast it is sometimes difﬁcult to communicate
the content or format of a task if no examples are given.
6

Language Models are Few-Shot Learners

40+32 pages


the content or format of a task if no examples are given.
6
Figure 2.1: Zero-shot, one-shot and few-shot, contrasted with traditional ﬁne-tuning . The panels above show
four methods for performing a task with a language model – ﬁne-tuning is the traditional method, whereas zero-, one-,
and few-shot, which we study in this work, require the model to perform the task with only forward passes at test
time. We typically present the model with a few dozen examples in the few shot setting. Exact phrasings for all task
descriptions, examples and prompts can be found in Appendix G.
•Zero-Shot (0S) is the same as one-shot except that no demonstrations are allowed, and the model is only given
a natural language instruction describing the task. This method provides maximum convenience, potential for
robustness, and avoidance of spurious correlations (unless they occur very broadly across the large corpus of
pre-training data), but is also the most challenging setting. In some cases it may even be difﬁcult for humans
to understand the format of the task without prior examples, so this setting is in some cases “unfairly hard”.
For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be

Language Models are Few-Shot Learners

40+32 pages


For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be
ambiguous, as it may not be clear exactly what format the table should have or what should be included (and
even with careful clariﬁcation, understanding precisely what is desired can be difﬁcult). Nevertheless, for at
least some settings zero-shot is closest to how humans perform tasks – for example, in the translation example
in Figure 2.1, a human would likely know what to do from just the text instruction.
Figure 2.1 shows the four methods using the example of translating English to French. In this paper we focus on
zero-shot, one-shot and few-shot, with the aim of comparing them not as competing alternatives, but as different
problem settings which offer a varying trade-off between performance on speciﬁc benchmarks and sample efﬁciency.
We especially highlight the few-shot results as many of them are only slightly behind state-of-the-art ﬁne-tuned models.
Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance,
and are important targets for future work.

Language Models are Few-Shot Learners

40+32 pages


Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance,
and are important targets for future work.
Sections 2.1-2.3 below give details on our models, training data, and training process respectively. Section 2.4 discusses
the details of how we do few-shot, one-shot, and zero-shot evaluations.
7
Model Name nparamsnlayersdmodelnheadsdhead Batch Size Learning Rate
GPT-3 Small 125M 12 768 12 64 0.5M 6:0104
GPT-3 Medium 350M 24 1024 16 64 0.5M 3:0104
GPT-3 Large 760M 24 1536 16 96 0.5M 2:5104
GPT-3 XL 1.3B 24 2048 24 128 1M 2:0104
GPT-3 2.7B 2.7B 32 2560 32 80 1M 1:6104
GPT-3 6.7B 6.7B 32 4096 32 128 2M 1:2104
GPT-3 13B 13.0B 40 5140 40 128 2M 1:0104

Language Models are Few-Shot Learners

40+32 pages


GPT-3 13B 13.0B 40 5140 40 128 2M 1:0104
GPT-3 175B or “GPT-3” 175.0B 96 12288 96 128 3.2M 0:6104
Table 2.1: Sizes, architectures, and learning hyper-parameters (batch size in tokens and learning rate) of the models
which we trained. All models were trained for a total of 300 billion tokens.
2.1 Model and Architectures
We use the same model and architecture as GPT-2 [ RWC+19], including the modiﬁed initialization, pre-normalization,
and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse
attention patterns in the layers of the transformer, similar to the Sparse Transformer [ CGRS19 ]. To study the dependence
of ML performance on model size, we train 8 different sizes of model, ranging over three orders of magnitude from 125
million parameters to 175 billion parameters, with the last being the model we call GPT-3. Previous work [ KMH+20]
suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a
function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for
downstream language tasks.

Language Models are Few-Shot Learners

40+32 pages


function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for
downstream language tasks.
Table 2.1 shows the sizes and architectures of our 8 models. Here nparams is the total number of trainable parameters,
nlayers is the total number of layers, dmodel is the number of units in each bottleneck layer (we always have the
feedforward layer four times the size of the bottleneck layer, d= 4dmodel ), anddhead is the dimension of each
attention head. All models use a context window of nctx= 2048 tokens. We partition the model across GPUs along
both the depth and width dimension in order to minimize data-transfer between nodes. The precise architectural
parameters for each model are chosen based on computational efﬁciency and load-balancing in the layout of models
across GPU’s. Previous work [ KMH+20] suggests that validation loss is not strongly sensitive to these parameters
within a reasonably broad range.
2.2 Training Dataset
Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset2[RSR+19] constituting
nearly a trillion words. This size of dataset is sufﬁcient to train our largest models without ever updating on the same

Language Models are Few-Shot Learners

40+32 pages


nearly a trillion words. This size of dataset is sufﬁcient to train our largest models without ever updating on the same
sequence twice. However, we have found that unﬁltered or lightly ﬁltered versions of Common Crawl tend to have
lower quality than more curated datasets. Therefore, we took 3 steps to improve the average quality of our datasets:
(1) we downloaded and ﬁltered a version of CommonCrawl based on similarity to a range of high-quality reference
corpora, (2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy
and preserve the integrity of our held-out validation set as an accurate measure of overﬁtting, and (3) we also added
known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.
Details of the ﬁrst two points (processing of Common Crawl) are described in Appendix A. For the third, we added
several curated high-quality datasets, including an expanded version of the WebText dataset [ RWC+19], collected
by scraping links over a longer period of time, and ﬁrst described in [ KMH+20], two internet-based books corpora
(Books1 and Books2) and English-language Wikipedia.

Language Models are Few-Shot Learners

40+32 pages


(Books1 and Books2) and English-language Wikipedia.
Table 2.2 shows the ﬁnal mixture of datasets that we used in training. The CommonCrawl data was downloaded from
41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before ﬁltering
and 570GB after ﬁltering, roughly equivalent to 400 billion byte-pair-encoded tokens. Note that during training, datasets
are not sampled in proportion to their size, but rather datasets we view as higher-quality are sampled more frequently,
such that CommonCrawl and Books2 datasets are sampled less than once during training, but the other datasets are
sampled 2-3 times. This essentially accepts a small amount of overﬁtting in exchange for higher quality training data.
2https://commoncrawl.org/the-data/
8
Figure 2.2: Total compute used during training . Based on the analysis in Scaling Laws For Neural Language Models
[KMH+20] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B
is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaﬂop/s-days of compute
during pre-training. Methodology for these calculations can be found in Appendix D.

Language Models are Few-Shot Learners

40+32 pages


during pre-training. Methodology for these calculations can be found in Appendix D.
DatasetQuantity
(tokens)Weight in
training mixEpochs elapsed when
training for 300B tokens
Common Crawl (ﬁltered) 410 billion 60% 0.44
WebText2 19 billion 22% 2.9
Books1 12 billion 8% 1.9
Books2 55 billion 8% 0.43
Wikipedia 3 billion 3% 3.4
Table 2.2: Datasets used to train GPT-3 . “Weight in training mix” refers to the fraction of examples during training
that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a
result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets
are seen less than once.
A major methodological concern with language models pretrained on a broad swath of internet data, particularly large
models with the capacity to memorize vast amounts of content, is potential contamination of downstream tasks by
having their test or development sets inadvertently seen during pre-training. To reduce such contamination, we searched
for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper.

Language Models are Few-Shot Learners

40+32 pages


for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper.
Unfortunately, a bug in the ﬁltering caused us to ignore some overlaps, and due to the cost of training it was not feasible
to retrain the model. In Section 4 we characterize the impact of the remaining overlaps, and in future work we will
more aggressively remove data contamination.
2.3 Training Process
As found in [ KMH+20,MKAT18 ], larger models can typically use a larger batch size, but require a smaller learning
rate. We measure the gradient noise scale during training and use it to guide our choice of batch size [ MKAT18 ]. Table
2.1 shows the parameter settings we used. To train the larger models without running out of memory, we use a mixture
of model parallelism within each matrix multiply and model parallelism across the layers of the network. All models
were trained on V100 GPU’s on part of a high-bandwidth cluster provided by Microsoft. Details of the training process
and hyperparameter settings are described in Appendix B.
9
2.4 Evaluation
For few-shot learning, we evaluate each example in the evaluation set by randomly drawing Kexamples from that
task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task. For LAMBADA and Storycloze

Language Models are Few-Shot Learners

40+32 pages


task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task. For LAMBADA and Storycloze
there is no supervised training set available so we draw conditioning examples from the development set and evaluate
on the test set. For Winograd (the original, not SuperGLUE version) there is only one dataset, so we draw conditioning
examples directly from it.
Kcan be any value from 0 to the maximum amount allowed by the model’s context window, which is nctx= 2048
for all models and typically ﬁts 10to100examples. Larger values of Kare usually but not always better, so when a
separate development and test set are available, we experiment with a few values of Kon the development set and then
run the best value on the test set. For some tasks (see Appendix G) we also use a natural language prompt in addition to
(or forK= 0, instead of) demonstrations.
On tasks that involve choosing one correct completion from several options (multiple choice), we provide Kexamples
of context plus correct completion, followed by one example of context only, and compare the LM likelihood of
each completion. For most tasks we compare the per-token likelihood (to normalize for length), however on a small

Language Models are Few-Shot Learners

40+32 pages


each completion. For most tasks we compare the per-token likelihood (to normalize for length), however on a small
number of datasets (ARC, OpenBookQA, and RACE) we gain additional beneﬁt as measured on the development set
by normalizing by the unconditional probability of each completion, by computingP(completionjcontext)
P(completionjanswer context), where
answer context is the string "Answer: " or"A: " and is used to prompt that the completion should be an answer
but is otherwise generic.
On tasks that involve binary classiﬁcation, we give the options more semantically meaningful names (e.g. “True” or
“False” rather than 0 or 1) and then treat the task like multiple choice; we also sometimes frame the task similar to what
is done by [RSR+19] (see Appendix G) for details.
On tasks with free-form completion, we use beam search with the same parameters as [ RSR+19]: a beam width of 4
and a length penalty of = 0:6. We score the model using F1 similarity score, BLEU, or exact match, depending on
what is standard for the dataset at hand.
Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-,

Language Models are Few-Shot Learners

40+32 pages


what is standard for the dataset at hand.
Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-,
and few-shot). When the test set is private, our model is often too large to ﬁt on the test server, so we report results on
the development set. We do submit to the test server on a small number of datasets (SuperGLUE, TriviaQA, PiQa)
where we were able to make submission work, and we submit only the 200B few-shot results, and report development
set results for everything else.
3 Results
In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6
additional extra-small models with as few as 100,000 parameters. As observed in [ KMH+20], language modeling
performance follows a power-law when making efﬁcient use of training compute. After extending this trend by two
more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these
improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will
see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a
broad spectrum of natural language tasks.

Language Models are Few-Shot Learners

40+32 pages


see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a
broad spectrum of natural language tasks.
Below, we evaluate the 8 models described in Section 2 (the 175 billion parameter parameter GPT-3 and 7 smaller
models) on a wide range of datasets. We group the datasets into 9 categories representing roughly similar tasks.
In Section 3.1 we evaluate on traditional language modeling tasks and tasks that are similar to language modeling,
such as Cloze tasks and sentence/paragraph completion tasks. In Section 3.2 we evaluate on “closed book” question
answering tasks: tasks which require using the information stored in the model’s parameters to answer general
knowledge questions. In Section 3.3 we evaluate the model’s ability to translate between languages (especially one-shot
and few-shot). In Section 3.4 we evaluate the model’s performance on Winograd Schema-like tasks. In Section 3.5 we
evaluate on datasets that involve commonsense reasoning or question answering. In Section 3.6 we evaluate on reading
comprehension tasks, in Section 3.7 we evaluate on the SuperGLUE benchmark suite, and in 3.8 we brieﬂy explore
NLI. Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities –

Language Models are Few-Shot Learners

40+32 pages


NLI. Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities –
these tasks focus on on-the-ﬂy reasoning, adaptation skills, or open-ended text synthesis. We evaluate all tasks in the
few-shot, one-shot, and zero-shot settings.
10
Figure 3.1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy
validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior
observed in [ KMH+20] continues for an additional two orders of magnitude with only small deviations from the
predicted curve. For this ﬁgure, we exclude embedding parameters from compute and parameter counts.
Setting PTB
SOTA (Zero-Shot) 35.8a
GPT-3 Zero-Shot 20.5
Table 3.1: Zero-shot results on PTB language modeling dataset. Many other common language modeling datasets
are omitted because they are derived from Wikipedia or other sources which are included in GPT-3’s training data.
a[RWC+19]
3.1 Language Modeling, Cloze, and Completion Tasks
In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks

Language Models are Few-Shot Learners

40+32 pages


In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks
that involve predicting a single word of interest, completing a sentence or paragraph, or choosing between possible
completions of a piece of text.
3.1.1 Language Modeling
We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [ MKM+94] dataset measured in [ RWC+19]. We omit
the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the
one-billion word benchmark due to a high fraction of the dataset being contained in our training set. PTB escapes these
issues due to predating the modern internet. Our largest model sets a new SOTA on PTB by a substantial margin of 15
points, achieving a perplexity of 20.50. Note that since PTB is a traditional language modeling dataset it does not have
a clear separation of examples to deﬁne one-shot or few-shot evaluation around, so we measure only zero-shot.
3.1.2 LAMBADA
The LAMBADA dataset [ PKL+16] tests the modeling of long-range dependencies in text – the model is asked to
predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the

Language Models are Few-Shot Learners

40+32 pages


predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the
continued scaling of language models is yielding diminishing returns on this difﬁcult benchmark. [ BHT+20] reﬂect on
the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([ SPP+19]
11
SettingLAMBADA
(acc)LAMBADA
(ppl)StoryCloze
(acc)HellaSwag
(acc)
SOTA 68.0a8.63b91.8c85.6d
GPT-3 Zero-Shot 76.2 3.00 83.2 78.9
GPT-3 One-Shot 72.5 3.35 84.7 78.1
GPT-3 Few-Shot 86.4 1.92 87.7 79.3
Table 3.2: Performance on cloze and completion tasks. GPT-3 signiﬁcantly improves SOTA on LAMBADA while
achieving respectable performance on two difﬁcult completion prediction datasets.a[Tur20 ]b[RWC+19]c[LDL19 ]
d[LCH+20]

Language Models are Few-Shot Learners

40+32 pages


d[LCH+20]
Figure 3.2: On LAMBADA, the few-shot capability of language models results in a strong boost to accuracy. GPT-3
2.7B outperforms the SOTA 17B parameter Turing-NLG [ Tur20 ] in this setting, and GPT-3 175B advances the state of
the art by 18%. Note zero-shot uses a different format from one-shot and few-shot as described in the text.
and [ Tur20 ]) and argue that “continuing to expand hardware and data sizes by orders of magnitude is not the path
forward”. We ﬁnd that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of
8% over the previous state of the art.
LAMBADA is also a demonstration of the ﬂexibility of few-shot learning as it provides a way to address a problem that
classically occurs with this dataset. Although the completion in LAMBADA is always the last word in a sentence, a
standard language model has no way of knowing this detail. It thus assigns probability not only to the correct ending but
also to other valid continuations of the paragraph. This problem has been partially addressed in the past with stop-word

Language Models are Few-Shot Learners

40+32 pages


also to other valid continuations of the paragraph. This problem has been partially addressed in the past with stop-word
ﬁlters [ RWC+19] (which ban “continuation” words). The few-shot setting instead allows us to “frame” the task as a
cloze-test and allows the language model to infer from examples that a completion of exactly one word is desired. We
use the following ﬁll-in-the-blank format:
Alice was friends with Bob. Alice went to visit her friend .!Bob
George bought some baseball equipment, a ball, a glove, and a .!
When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase
of over 18% from the previous state-of-the-art. We observe that few-shot performance improves strongly with model
size. While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy
by 10%. Finally, the ﬁll-in-blank method is not effective one-shot, where it always performs worse than the zero-shot
setting. Perhaps this is because all models still require several examples to recognize the pattern.
12
Setting NaturalQS WebQS TriviaQA

Language Models are Few-Shot Learners

40+32 pages


setting. Perhaps this is because all models still require several examples to recognize the pattern.
12
Setting NaturalQS WebQS TriviaQA
RAG (Fine-tuned, Open-Domain) [LPP+20] 44.5 45.5 68.0
T5-11B+SSM (Fine-tuned, Closed-Book) [RRS20] 36.6 44.7 60.5
T5-11B (Fine-tuned, Closed-Book) 34.5 37.4 50.1
GPT-3 Zero-Shot 14.6 14.4 64.3
GPT-3 One-Shot 23.0 25.3 68.0
GPT-3 Few-Shot 29.9 41.5 71.2
Table 3.3: Results on three Open-Domain QA tasks. GPT-3 is shown in the few-, one-, and zero-shot settings, as
compared to prior SOTA results for closed book and open domain settings. TriviaQA few-shot result is evaluated on the
wiki split test server.
One note of caution is that an analysis of test set contamination identiﬁed that a signiﬁcant minority of the LAMBADA
dataset appears to be present in our training data – however analysis performed in Section 4 suggests negligible impact
on performance.

Language Models are Few-Shot Learners

40+32 pages


dataset appears to be present in our training data – however analysis performed in Section 4 suggests negligible impact
on performance.
3.1.3 HellaSwag
The HellaSwag dataset [ ZHB+19] involves picking the best ending to a story or set of instructions. The examples were
adversarially mined to be difﬁcult for language models while remaining easy for humans (who achieve 95.6% accuracy).
GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the
75.4% accuracy of a ﬁne-tuned 1.5B parameter language model [ ZHR+19] but still a fair amount lower than the overall
SOTA of 85.6% achieved by the ﬁne-tuned multi-task model ALUM.
3.1.4 StoryCloze
We next evaluate GPT-3 on the StoryCloze 2016 dataset [ MCH+16], which involves selecting the correct ending
sentence for ﬁve-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot

Language Models are Few-Shot Learners

40+32 pages


sentence for ﬁve-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot
setting (with K= 70 ). This is still 4.1% lower than the ﬁne-tuned SOTA using a BERT based model [ LDL19 ] but
improves over previous zero-shot results by roughly 10%.
3.2 Closed Book Question Answering
In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense
amount of possible queries, this task has normally been approached by using an information retrieval system to ﬁnd
relevant text in combination with a model which learns to generate an answer given the question and the retrieved
text. Since this setting allows a system to search for and condition on text which potentially contains the answer it
is denoted “open-book”. [ RRS20 ] recently demonstrated that a large language model can perform surprisingly well
directly answering the questions without conditioning on auxilliary information. They denote this more restrictive
evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better

Language Models are Few-Shot Learners

40+32 pages


evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better
and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [ RRS20 ]: Natural Questions [ KPR+19],
WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in
the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than
previous closed-book QA work: in addition to external content not being allowed, ﬁne-tuning on the Q&A dataset itself
is also not permitted.
The results for GPT-3 are shown in Table 3.3. On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the
one-shot setting, and 71.2% in the few-shot setting. The zero-shot result already outperforms the ﬁne-tuned T5-11B by
14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%. The one-shot

Language Models are Few-Shot Learners

40+32 pages


14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%. The one-shot
result improves by 3.7% and matches the SOTA for an open-domain QA system which not only ﬁne-tunes but also
makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [ LPP+20].
GPT-3’s few-shot result further improves performance another 3.2% beyond this.
On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5%
in the few-shot setting. This compares to 37.4% for ﬁne-tuned T5-11B, and 44.7% for ﬁne-tuned T5-11B+SSM,
which uses a Q&A-speciﬁc pre-training procedure. GPT-3 in the few-shot setting approaches the performance of
state-of-the-art ﬁne-tuned models. Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to

Language Models are Few-Shot Learners

40+32 pages


state-of-the-art ﬁne-tuned models. Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to
few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions
13
Figure 3.3: On TriviaQA GPT3’s performance grows smoothly with model size, suggesting that language models
continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make signiﬁcant gains
over zero-shot behavior, matching and exceeding the performance of the SOTA ﬁne-tuned open-domain model, RAG
[LPP+20]
and/or the style of their answers are out-of-distribution for GPT-3. Nevertheless, GPT-3 appears able to adapt to this
distribution, recovering strong performance in the few-shot setting.
On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in
the few-shot setting, compared to 36.6% for ﬁne-tuned T5 11B+SSM. Similar to WebQS, the large gain from zero-shot

Language Models are Few-Shot Learners

40+32 pages


to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to
TriviaQA and WebQS. In particular, the questions in NQs tend towards very ﬁne-grained knowledge on Wikipedia
speciﬁcally which could be testing the limits of GPT-3’s capacity and broad pretraining distribution.
Overall, on one of the three datasets GPT-3’s one-shot matches the open-domain ﬁne-tuning SOTA. On the other two
datasets it approaches the performance of the closed-book SOTA despite not using ﬁne-tuning. On all 3 datasets, we
ﬁnd that performance scales very smoothly with model size (Figure 3.3 and Appendix H Figure H.7), possibly reﬂecting
the idea that model capacity translates directly to more ‘knowledge’ absorbed in the parameters of the model.
3.3 Translation
For GPT-2 a ﬁlter was used on a multilingual collection of documents to produce an English only dataset due to capacity
concerns. Even with this ﬁltering GPT-2 showed some evidence of multilingual capability and performed non-trivially
when translating between French and English despite only training on 10 megabytes of remaining French text. Since we

Language Models are Few-Shot Learners

40+32 pages


when translating between French and English despite only training on 10 megabytes of remaining French text. Since we
increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training
dataset to include more representation of other languages, though this remains an area for further improvement. As
discussed in 2.2 the majority of our data is derived from raw Common Crawl with only quality-based ﬁltering. Although
GPT-3’s training data is still primarily English (93% by word count), it also includes 7% of text in other languages.
These languages are documented in the supplemental material. In order to better understand translation capability, we
also expand our analysis to include two additional commonly studied languages, German and Romanian.
Existing unsupervised machine translation approaches often combine pretraining on a pair of monolingual datasets
with back-translation [ SHB15 ] to bridge the two languages in a controlled way. By contrast, GPT-3 learns from a
blend of training data that mixes many languages together in a natural way, combining them on a word, sentence,
and document level. GPT-3 also uses a single training objective which is not customized or designed for any task in
particular. However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make

Language Models are Few-Shot Learners

40+32 pages


particular. However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make
use of a small amount of paired examples (1 or 64). This corresponds to up to a page or two of in-context training data.
Results are shown in Table 3.4. Zero-shot GPT-3, which only receives on a natural language description of the task,
still underperforms recent unsupervised NMT results. However, providing only a single example demonstration for
14
Setting En !Fr Fr!En En!De De!En En!Ro Ro!En
SOTA (Supervised) 45.6a35.0b41.2c40.2d38.5e39.9e
XLM [LC19] 33.4 33.3 26.4 34.3 33.3 31.8
MASS [STQ+19] 37.5 34.9 28.3 35.2 35.2 33.1
mBART [LGG+20] - - 29.8 34.0 35.0 30.5
GPT-3 Zero-Shot 25.2 21.2 24.6 27.2 14.1 19.9
GPT-3 One-Shot 28.3 33.7 26.2 30.4 20.6 38.6

Language Models are Few-Shot Learners

40+32 pages


GPT-3 One-Shot 28.3 33.7 26.2 30.4 20.6 38.6
GPT-3 Few-Shot 32.6 39.2 29.7 40.6 21.0 39.5
Table 3.4: Few-shot GPT-3 outperforms previous unsupervised NMT work by 5 BLEU when translating
into English reﬂecting its strength as an English LM. We report BLEU scores on the WMT’14 Fr $En,
WMT’16 De$En, and WMT’16 Ro $En datasets as measured by multi-bleu.perl with XLM’s tokenization in order to compare most closely with prior unsupervised NMT work. SacreBLEUf[Pos18 ] results reported in Appendix H. Underline indicates an unsupervised or few-shot SOTA, bold indicates supervised SOTA
with relative conﬁdence.a[EOAG18 ]b[DHKH14 ]c[WXH+18]d[oR16 ]e[LGG+20]f[SacreBLEU signature:
BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]

Language Models are Few-Shot Learners

40+32 pages


BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]
Figure 3.4: Few-shot translation performance on 6 language pairs as model capacity increases. There is a consistent
trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be
stronger than translation from English.
15
Setting Winograd Winogrande (XL)
Fine-tuned SOTA 90.1a84.6b
GPT-3 Zero-Shot 88.3* 70.2
GPT-3 One-Shot 89.7* 73.2
GPT-3 Few-Shot 88.6* 77.7
Table 3.5: Results on the WSC273 version of Winograd schemas and the adversarial Winogrande dataset. See Section
4 for details on potential contamination of the Winograd test set.a[SBBC19]b[LYN+20]
Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales.
Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B
is competitive with a ﬁne-tuned RoBERTA-large.

Language Models are Few-Shot Learners

40+32 pages


is competitive with a ﬁne-tuned RoBERTA-large.
each translation task improves performance by over 7 BLEU and nears competitive performance with prior work.
GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior
unsupervised NMT work. GPT-3 has a noticeable skew in its performance depending on language direction. For the
three input languages studied, GPT-3 signiﬁcantly outperforms prior unsupervised NMT work when translating into
English but underperforms when translating in the other direction. Performance on En-Ro is a noticeable outlier at
over 10 BLEU worse than prior unsupervised NMT work. This could be a weakness due to reusing the byte-level BPE
tokenizer of GPT-2 which was developed for an almost entirely English training dataset. For both Fr-En and De-En,
few shot GPT-3 outperforms the best supervised result we could ﬁnd but due to our unfamiliarity with the literature and
the appearance that these are un-competitive benchmarks we do not suspect those results represent true state of the art.
For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of

Language Models are Few-Shot Learners

40+32 pages


For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of
unsupervised pretraining, supervised ﬁnetuning on 608K labeled examples, and backtranslation [LHCG19b].
Finally, across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of
improvement with model capacity. This is shown in Figure 3.4 in the case of few-shot results, and scaling for all three
settings is shown in Appendix H.
3.4 Winograd-Style Tasks
The Winograd Schemas Challenge [ LDM12 ] is a classical task in NLP that involves determining which word a pronoun
refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human. Recently ﬁne-tuned
language models have achieved near-human performance on the original Winograd dataset, but more difﬁcult versions
16
Setting PIQA ARC (Easy) ARC (Challenge) OpenBookQA
Fine-tuned SOTA 79.4 92.0[KKS+20] 78.5[KKS+20] 87.2[KKS+20]
GPT-3 Zero-Shot 80.5* 68.8 51.4 57.6

Language Models are Few-Shot Learners

40+32 pages


GPT-3 Zero-Shot 80.5* 68.8 51.4 57.6
GPT-3 One-Shot 80.5* 71.2 53.2 58.8
GPT-3 Few-Shot 82.8* 70.1 51.5 65.4
Table 3.6: GPT-3 results on three commonsense reasoning tasks, PIQA, ARC, and OpenBookQA. GPT-3 Few-Shot
PIQA result is evaluated on the test server. See Section 4 for details on potential contamination issues on the PIQA test
set.
Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings. The largest model achieves a
score on the development set in all three conditions that exceeds the best recorded score on the task.
such as the adversarially-mined Winogrande dataset [ SBBC19 ] still signiﬁcantly lag human performance. We test
GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.
On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method

Language Models are Few-Shot Learners

40+32 pages


On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method
described in [ RWC+19]. Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which
is presented as binary classiﬁcation and requires entity extraction to convert to the form described in this section. On
Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear
in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human
performance. We note that contamination analysis found some Winograd schemas in the training data but this appears
to have only a small effect on results (see Section 4).
On the more difﬁcult Winogrande dataset, we do ﬁnd gains to in-context learning: GPT-3 achieves 70.2% in the
zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a ﬁne-tuned

Language Models are Few-Shot Learners

40+32 pages


zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a ﬁne-tuned
RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a ﬁne-tuned high capacity model (T5), and
human performance on the task as reported by [SBBC19] is 94.0%.
3.5 Common Sense Reasoning
Next we consider three datasets which attempt to capture physical or scientiﬁc reasoning, as distinct from sentence
completion, reading comprehension, or broad knowledge question answering. The ﬁrst, PhysicalQA (PIQA) [ BZB+19],
asks common sense questions about how the physical world works and is intended as a probe of grounded understanding
of the world. GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot
(the last measured on PIQA’s test server). This compares favorably to the 79.4% accuracy prior state-of-the-art of a
17
Setting CoQA DROP QuAC SQuADv2 RACE-h RACE-m

Language Models are Few-Shot Learners

40+32 pages


17
Setting CoQA DROP QuAC SQuADv2 RACE-h RACE-m
Fine-tuned SOTA 90.7a89.1b74.4c93.0d90.0e93.1e
GPT-3 Zero-Shot 81.5 23.6 41.5 59.5 45.5 58.4
GPT-3 One-Shot 84.0 34.3 43.3 65.4 45.9 57.4
GPT-3 Few-Shot 85.0 36.5 44.3 69.8 46.8 58.1
Table 3.7: Results on reading comprehension tasks. All scores are F1 except results for RACE which report accuracy.
a[JZC+19]b[JN20]c[AI19]d[QIA20]e[SPP+19]
ﬁne-tuned RoBERTa. PIQA shows relatively shallow scaling with model size and is still over 10% worse than human
performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art. Our analysis
ﬂagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark
the result with an asterisk. See Section 4 for details.

Language Models are Few-Shot Learners

40+32 pages


ﬂagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark
the result with an asterisk. See Section 4 for details.
ARC [ CCE+18] is a dataset of multiple-choice questions collected from 3rd to 9th grade science exams. On the
“Challenge” version of the dataset which has been ﬁltered to questions which simple statistical or information retrieval
methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot
setting, and 51.5% in the few-shot setting. This is approaching the performance of a ﬁne-tuned RoBERTa baseline
(55.9%) from UniﬁedQA [ KKS+20]. On the “Easy” version of the dataset (questions which either of the mentioned
baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a ﬁne-tuned
RoBERTa baseline from [ KKS+20]. However, both of these results are still much worse than the overall SOTAs

Language Models are Few-Shot Learners

40+32 pages


RoBERTa baseline from [ KKS+20]. However, both of these results are still much worse than the overall SOTAs
achieved by the UniﬁedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% on the easy
set.
On OpenBookQA [ MCKS18 ], GPT-3 improves signiﬁcantly from zero to few shot settings but is still over 20 points
short of the overall SOTA. GPT-3’s few-shot performance is similar to a ﬁne-tuned BERT Large baseline on the
leaderboard.
Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and
inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a signiﬁcant
improvement is observed on OpenBookQA. GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings.
3.6 Reading Comprehension
Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive,
multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread

Language Models are Few-Shot Learners

40+32 pages


multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread
in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general
we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each
respective dataset.
GPT-3 performs best (within 3 points of the human baseline) on CoQA [ RCM19 ] a free-form conversational dataset
and performs worst (13 F1 below an ELMo baseline) on QuAC [ CHI+18] a dataset which requires modeling structured
dialog acts and answer span selections of teacher-student interactions. On DROP [ DWD+19], a dataset testing discrete
reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the ﬁne-tuned
BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches
which augment neural networks with symbolic systems [ RLL+19]. On SQuAD 2.0 [ RJL18 ], GPT-3 demonstrates its
few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to

Language Models are Few-Shot Learners

40+32 pages


few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to
slightly outperform the best ﬁne-tuned result in the original paper. On RACE [ LXL+17], a multiple choice dataset of
middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with
the earliest work utilizing contextual representations and is still 45% behind SOTA.
3.7 SuperGLUE
In order to better aggregate results on NLP tasks and compare to popular models such as BERT and RoBERTa in a
more systematic way, we also evaluate GPT-3 on a standardized collection of datasets, the SuperGLUE benchmark
[WPN+19] [WPN+19] [CLC+19] [DMST19 ] [RBG11 ] [KCR+18] [ZLL+18] [DGM06 ] [BHDD+06] [GMDD07 ]
[BDD+09] [PCC18 ] [PHR+18]. GPT-3’s test-set performance on the SuperGLUE dataset is shown in Table 3.8. In the
few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC
18

Language Models are Few-Shot Learners

40+32 pages


few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC
18
Figure 3.7: GPT-3 results on CoQA reading comprehension task. GPT-3 175B achieves 85 F1 in the few-shot setting,
only a few points behind measured human performance and state-of-the-art ﬁne-tuned models. Zero-shot and one-shot
performance is a few points behind, with the gains to few-shot being largest for bigger models.
SuperGLUE BoolQ CB CB COPA RTE
Average Accuracy Accuracy F1 Accuracy Accuracy
Fine-tuned SOTA 89.0 91.0 96.9 93.9 94.8 92.5
Fine-tuned BERT-Large 69.0 77.4 83.6 75.7 70.6 71.7
GPT-3 Few-Shot 71.8 76.4 75.6 52.0 92.0 69.0
WiC WSC MultiRC MultiRC ReCoRD ReCoRD
Accuracy Accuracy Accuracy F1a Accuracy F1
Fine-tuned SOTA 76.1 93.8 62.3 88.2 92.5 93.3
Fine-tuned BERT-Large 69.6 64.6 24.1 70.0 71.3 72.0

Language Models are Few-Shot Learners

40+32 pages


Fine-tuned BERT-Large 69.6 64.6 24.1 70.0 71.3 72.0
GPT-3 Few-Shot 49.4 80.1 30.5 75.4 90.2 91.1
Table 3.8: Performance of GPT-3 on SuperGLUE compared to ﬁne-tuned baselines and SOTA. All results are reported
on the test set. GPT-3 few-shot is given a total of 32 examples within the context of each task and performs no gradient
updates.
19
Figure 3.8: Performance on SuperGLUE increases with model size and number of examples in context. A value
ofK= 32 means that our model was shown 32 examples per task, for 256 examples total divided across the 8 tasks in
SuperGLUE. We report GPT-3 values on the dev set, so our numbers are not directly comparable to the dotted reference
lines (our test set results are in Table 3.8). The BERT-Large reference model was ﬁne-tuned on the SuperGLUE training
set (125K examples), whereas BERT++ was ﬁrst ﬁne-tuned on MultiNLI (392K examples) and SWAG (113K examples)

Language Models are Few-Shot Learners

40+32 pages