set (125K examples), whereas BERT++ was ﬁrst ﬁne-tuned on MultiNLI (392K examples) and SWAG (113K examples)
before further ﬁne-tuning on the SuperGLUE training set (for a total of 630K ﬁne-tuning examples). We ﬁnd the
difference in performance between the BERT-Large and BERT++ to be roughly equivalent to the difference between
GPT-3 with one example per context versus eight examples per context.
and MultiRC, we sampled a new set of examples to use in the context for each problem. For WSC and MultiRC, we
used the same set of randomly drawn examples from the training set as context for all of the problems we evaluated.
We observe a wide range in GPT-3’s performance across tasks. On COPA and ReCoRD GPT-3 achieves near-SOTA
performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving
second place on the leaderboard, where ﬁrst place is held by a ﬁne-tuned 11 billion parameter model (T5). On WSC,
performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the

Language Models are Few-Shot Learners

40+32 pages


performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the
original Winograd dataset as described in Section 3.4). On BoolQ, MultiRC, and RTE, performance is reasonable,
roughly matching that of a ﬁne-tuned BERT-Large. On CB, we see signs of life at 75.6% in the few-shot setting.
WiC is a notable weak spot with few-shot performance at 49.4% (at random chance). We tried a number of different
phrasings and formulations for WiC (which involves determining if a word is being used with the same meaning in two
sentences), none of which was able to achieve strong performance. This hints at a phenomenon that will become clearer
in the next section (which discusses the ANLI benchmark) – GPT-3 appears to be weak in the few-shot or one-shot
setting at some tasks that involve comparing two sentences or snippets, for example whether a word is used the same
way in two sentences (WiC), whether one sentence is a paraphrase of another, or whether one sentence implies another.
This could also explain the comparatively low scores for RTE and CB, which also follow this format. Despite these

Language Models are Few-Shot Learners

Summary:
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - something which
current NLP systems still largely struggle to do. Here we show that scaling up
language models greatly improves task-agnostic, few-shot performance, sometimes
even reaching competitiveness with prior state-of-the-art fine-tuning
approaches. Specifically, we train GPT-3, an autoregressive language model with
175 billion parameters, 10x more than any previous non-sparse language model,
and test its performance in the few-shot setting. For all tasks, GPT-3 is
applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require
on-the-fly reasoning or domain adaptation, such as unscrambling words, using a
novel word in a sentence, or performing 3-digit arithmetic. At the same time,
we also identify some datasets where GPT-3's few-shot learning still struggles,
as well as some datasets where GPT-3 faces methodological issues related to
training on large web corpora. Finally, we find that GPT-3 can generate samples
of news articles which human evaluators have difficulty distinguishing from
articles written by humans. We discuss broader societal impacts of this finding
and of GPT-3 in general.

Tom B. Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell
Sandhini Agarwal
Ariel Herbert-Voss
Gretchen Krueger
Tom Henighan
Rewon Child
Aditya Ramesh
Daniel M. Ziegler
Jeffrey Wu
Clemens Winter
Christopher Hesse
Mark Chen
Eric Sigler
Mateusz Litwin
Scott Gray
Benjamin Chess
Jack Clark
Christopher Berner
Sam McCandlish
Alec Radford
Ilya Sutskever
Dario Amodei

40+32 pages

Learning to learn by gradient descent by gradient descent

Massively multilingual neural machine translation

Roee Aharoni, Melvin Johnson, Orhan Firat

Language (technology) is power: A critical survey of “bias” in NLP


This could also explain the comparatively low scores for RTE and CB, which also follow this format. Despite these
weaknesses, GPT-3 still outperforms a ﬁne-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to
the state-of-the-art held by a ﬁne-tuned 11 billion parameter model.
Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of
examples in the context showing increasing beneﬁts from in-context learning (Figure 3.8). We scale Kup to 32
examples per task, after which point additional examples will not reliably ﬁt into our context. When sweeping over
values ofK, we ﬁnd that GPT-3 requires less than eight total examples per task to outperform a ﬁne-tuned BERT-Large
on overall SuperGLUE score.
3.8 NLI
Natural Language Inference (NLI) [ Fyo00 ] concerns the ability to understand the relationship between two sentences.
In practice, this task is usually structured as a two or three class classiﬁcation problem where the model classiﬁes
20

Language Models are Few-Shot Learners

40+32 pages


In practice, this task is usually structured as a two or three class classiﬁcation problem where the model classiﬁes
20
Figure 3.9: Performance of GPT-3 on ANLI Round 3. Results are on the dev-set, which has only 1500 examples
and therefore has high variance (we estimate a standard deviation of 1.2%). We ﬁnd that smaller models hover around
random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA. Results for
ANLI rounds 1 and 2 are shown in the appendix.
whether the second sentence logically follows from the ﬁrst, contradicts the ﬁrst sentence, or is possibly true (neutral).
SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task. On RTE, only the largest
version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting
GPT-3 performs similarly to a single-task ﬁne-tuned BERT Large. We also evaluate on the recently introduced
Adversarial Natural Language Inference (ANLI) dataset [ NWD+19]. ANLI is a difﬁcult dataset employing a series of

Language Models are Few-Shot Learners

40+32 pages


Adversarial Natural Language Inference (ANLI) dataset [ NWD+19]. ANLI is a difﬁcult dataset employing a series of
adversarially mined natural language inference questions in three rounds (R1, R2, and R3). Similar to RTE, all of our
models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting ( 33%),
whereas GPT-3 itself shows signs of life on Round 3. Results for ANLI R3 are highlighted in Figure 3.9 and full results
for all rounds can be found in Appendix H. These results on both RTE and ANLI suggest that NLI is still a very difﬁcult
task for language models and they are only just beginning to show signs of progress.
3.9 Synthetic and Qualitative Tasks
One way to probe GPT-3’s range of abilities in the few-shot (or zero- and one-shot) setting is to give it tasks which
require it to perform simple on-the-ﬂy computational reasoning, recognize a novel pattern that is unlikely to have
occurred in training, or adapt quickly to an unusual task. We devise several tasks to test this class of abilities. First, we

Language Models are Few-Shot Learners

40+32 pages


occurred in training, or adapt quickly to an unusual task. We devise several tasks to test this class of abilities. First, we
test GPT-3’s ability to perform arithmetic. Second, we create several tasks that involve rearranging or unscrambling the
letters in a word, tasks which are unlikely to have been exactly seen during training. Third, we test GPT-3’s ability to
solve SAT-style analogy problems few-shot. Finally, we test GPT-3 on several qualitative tasks, including using new
words in a sentence, correcting English grammar, and news article generation. We will release the synthetic datasets
with the hope of stimulating further study of test-time behavior of language models.
3.9.1 Arithmetic
To test GPT-3’s ability to perform simple arithmetic operations without task-speciﬁc training, we developed a small
battery of 10 tests that involve asking GPT-3 a simple arithmetic problem in natural language:
•2 digit addition (2D+) – The model is asked to add two integers sampled uniformly from [0;100) , phrased in
the form of a question, e.g. “Q: What is 48 plus 76? A: 124.”
•2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0;100) ; the

Language Models are Few-Shot Learners

40+32 pages


•2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0;100) ; the
answer may be negative. Example: “Q: What is 34 minus 53? A: -19”.
•3 digit addition (3D+) – Same as 2 digit addition, except numbers are uniformly sampled from [0;1000) .
21
Figure 3.10: Results on all 10 arithmetic tasks in the few-shot settings for models of different sizes. There is a
signiﬁcant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being
able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a signiﬁcant fraction
of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations. Results for one-shot and zero-shot
are shown in the appendix.
•3 digit subtraction (3D-) – Same as 2 digit subtraction, except numbers are uniformly sampled from [0;1000) .
•4 digit addition (4D+) – Same as 3 digit addition, except uniformly sampled from [0;10000) .
•4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0;10000) .

Language Models are Few-Shot Learners

40+32 pages


•4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0;10000) .
•5 digit addition (5D+) – Same as 3 digit addition, except uniformly sampled from [0;100000) .
•5 digit subtraction (5D-) – Same as 3 digit subtraction, except uniformly sampled from [0;100000) .
•2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from [0;100) ,
e.g. “Q: What is 24 times 42? A: 1008”.
•One-digit composite (1DC) – The model is asked to perform a composite operation on three 1 digit numbers,
with parentheses around the last two. For example, “Q: What is 6+(4*8)? A: 38”. The three 1 digit numbers
are selected uniformly on [0;10)and the operations are selected uniformly from f+,-,*g.
In all 10 tasks the model must generate the correct answer exactly. For each task we generate a dataset of 2,000 random
instances of the task and evaluate all models on those instances.
First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10. On addition and subtraction,

Language Models are Few-Shot Learners

40+32 pages


First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10. On addition and subtraction,
GPT-3 displays strong proﬁciency when the number of digits is small, achieving 100% accuracy on 2 digit addition,
98.9% at 2 digit subtraction, 80.2% at 3 digit addition, and 94.2% at 3-digit subtraction. Performance decreases as the
number of digits increases, but GPT-3 still achieves 25-26% accuracy on four digit operations and 9-10% accuracy on
ﬁve digit operations, suggesting at least some capacity to generalize to larger numbers of digits. GPT-3 also achieves
29.2% accuracy at 2 digit multiplication, an especially computationally intensive operation. Finally, GPT-3 achieves
21.3% accuracy at single digit combined operations (for example, 9*(7+5)), suggesting that it has some robustness
beyond just single operations.
As Figure 3.10 makes clear, small models do poorly on all of these tasks – even the 13 billion parameter model (the
second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all
other operations less than 10% of the time.

Language Models are Few-Shot Learners

40+32 pages


second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all
other operations less than 10% of the time.
One-shot and zero-shot performance are somewhat degraded relative to few-shot performance, suggesting that adaptation
to the task (or at the very least recognition of the task) is important to performing these computations correctly.
Nevertheless, one-shot performance is still quite strong, and even zero-shot performance of the full GPT-3 signiﬁcantly
22
Setting 2D+ 2D- 3D+ 3D- 4D+ 4D- 5D+ 5D- 2Dx 1DC
GPT-3 Zero-shot 76.9 58.0 34.2 48.3 4.0 7.5 0.7 0.8 19.8 9.8
GPT-3 One-shot 99.6 86.4 65.5 78.7 14.0 14.0 3.5 3.8 27.4 14.3
GPT-3 Few-shot 100.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3

Language Models are Few-Shot Learners

40+32 pages


GPT-3 Few-shot 100.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3
Table 3.9: Results on basic arithmetic tasks for GPT-3 175B. f2,3,4,5gDf+,-gis 2, 3, 4, and 5 digit addition or
subtraction, 2Dx is 2 digit multiplication. 1DC is 1 digit composite operations. Results become progressively stronger
moving from the zero-shot to one-shot to few-shot setting, but even the zero-shot shows signiﬁcant arithmetic abilities.
Setting CL A1 A2 RI RW
GPT-3 Zero-shot 3.66 2.28 8.91 8.26 0.09
GPT-3 One-shot 21.7 8.62 25.9 45.4 0.48
GPT-3 Few-shot 37.9 15.1 39.7 67.2 0.44
Table 3.10: GPT-3 175B performance on various word unscrambling and word manipulation tasks, in zero-, one-, and
few-shot settings. CL is “cycle letters in word”, A1 is anagrams of but the ﬁrst and last letters, A2 is anagrams of all but

Language Models are Few-Shot Learners

40+32 pages


the ﬁrst and last two letters, RI is “Random insertion in word”, RW is “reversed words”.
outperforms few-shot learning for all smaller models. All three settings for the full GPT-3 are shown in Table 3.9, and
model capacity scaling for all three settings is shown in Appendix H.
To spot-check whether the model is simply memorizing speciﬁc arithmetic problems, we took the 3-digit arithmetic
problems in our test set and searched for them in our training data in both the forms "<NUM1> + <NUM2> =" and
"<NUM1> plus <NUM2>" . Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000
subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers
could have been memorized. In addition, inspection of incorrect answers reveals that the model often makes mistakes
such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than
memorizing a table.
Overall, GPT-3 displays reasonable proﬁciency at moderately complex arithmetic in few-shot, one-shot, and even
zero-shot settings.

Language Models are Few-Shot Learners

40+32 pages


Overall, GPT-3 displays reasonable proﬁciency at moderately complex arithmetic in few-shot, one-shot, and even
zero-shot settings.
3.9.2 Word Scrambling and Manipulation Tasks
To test GPT-3’s ability to learn novel symbolic manipulations from a few examples, we designed a small battery of
5 “character manipulation” tasks. Each task involves giving the model a word distorted by some combination of
scrambling, addition, or deletion of characters, and asking it to recover the original word. The 5 tasks are:
•Cycle letters in word (CL) – The model is given a word with its letters cycled, then the “=” symbol, and
is expected to generate the original word. For example, it might be given “lyinevitab” and should output
“inevitably”.
•Anagrams of all but ﬁrst and last characters (A1) – The model is given a word where every letter except
the ﬁrst and last have been scrambled randomly, and must output the original word. Example: criroptuon =
corruption.
•Anagrams of all but ﬁrst and last 2 characters (A2) – The model is given a word where every letter except

Language Models are Few-Shot Learners

40+32 pages


corruption.
•Anagrams of all but ﬁrst and last 2 characters (A2) – The model is given a word where every letter except
the ﬁrst 2 and last 2 have been scrambled randomly, and must recover the original word. Example: opoepnnt
!opponent.
•Random insertion in word (RI) – A random punctuation or space character is inserted between each letter
of a word, and the model must output the original word. Example: s.u!c/c!e.s s i/o/n = succession.
•Reversed words (RW) – The model is given a word spelled backwards, and must output the original word.
Example: stcejbo!objects.
For each task we generate 10,000 examples, which we chose to be the top 10,000 most frequent words as measured by
[Nor09 ] of length more than 4 characters and less than 15 characters. The few-shot results are shown in Figure 3.11.
Task performance tends to grow smoothly with model size, with the full GPT-3 model achieving 66.9% on removing
23
Figure 3.11: Few-shot performance on the ﬁve word scrambling tasks for different sizes of model. There is generally
smooth improvement with model size although the random insertion task shows an upward slope of improvement with

Language Models are Few-Shot Learners

40+32 pages


smooth improvement with model size although the random insertion task shows an upward slope of improvement with
the 175B model solving the task the majority of the time. Scaling of one-shot and zero-shot performance is shown in
the appendix. All tasks are done with K= 100 .
random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task, and 15.1% on the more difﬁcult anagram
task (where only the ﬁrst and last letters are held ﬁxed). None of the models can reverse the letters in a word.
In the one-shot setting, performance is signiﬁcantly weaker (dropping by half or more), and in the zero-shot setting the
model can rarely perform any of the tasks (Table 3.10). This suggests that the model really does appear to learn these
tasks at test time, as the model cannot perform them zero-shot and their artiﬁcial nature makes them unlikely to appear
in the pre-training data (although we cannot conﬁrm this with certainty).
We can further quantify performance by plotting “in-context learning curves”, which show task performance as a
function of the number of in-context examples. We show in-context learning curves for the Symbol Insertion task

Language Models are Few-Shot Learners

40+32 pages


function of the number of in-context examples. We show in-context learning curves for the Symbol Insertion task
in Figure 1.2. We can see that larger models are able to make increasingly effective use of in-context information,
including both task examples and natural language task descriptions.
Finally, it is worth adding that solving these tasks requires character-level manipulations, whereas our BPE encoding
operates on signiﬁcant fractions of a word (on average 0:7words per token), so from the LM’s perspective succeeding
at these tasks involves not just manipulating BPE tokens but understanding and pulling apart their substructure. Also,
CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word),
requiring the model to perform some search to ﬁnd the correct unscrambling. Thus, the skills involved appear to require
non-trivial pattern-matching and computation.
3.9.3 SAT Analogies
To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of
374 “SAT analogy” problems [ TLBS03 ]. Analogies are a style of multiple choice question that constituted a section of

Language Models are Few-Shot Learners

40+32 pages


374 “SAT analogy” problems [ TLBS03 ]. Analogies are a style of multiple choice question that constituted a section of
the SAT college entrance exam before 2005. A typical example is “audacious is to boldness as (a) sanctimonious is to
hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to
temptation”. The student is expected to choose which of the ﬁve word pairs has the same relationship as the original
word pair; in this example the answer is “sanctimonious is to hypocrisy”. On this task GPT-3 achieves 65.2% in the
few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among
college applicants was 57% [ TL05 ] (random guessing yields 20%). As shown in Figure 3.12, the results improve with
scale, with the the full 175 billion model improving by over 10% compared to the 13 billion parameter model.
24
Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest

Language Models are Few-Shot Learners

40+32 pages


24
Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest
model achieves 65% accuracy in the few-shot setting, and also demonstrates signiﬁcant gains to in-context learning
which are not present in smaller models.
3.9.4 News Article Generation
Previous work on generative language models qualitatively tested their ability to generate synthetic “news articles” by
conditional sampling from the model given a human-written prompt consisting of a plausible ﬁrst sentence for a news
story [ RWC+19]. Relative to [ RWC+19], the dataset used to train GPT-3 is much less weighted towards news articles,
so trying to generate news articles via raw unconditional samples is less effective – for example GPT-3 often interprets
the proposed ﬁrst sentence of a “news article” as a tweet and then posts synthetic responses or follow-up tweets. To
solve this problem we employed GPT-3’s few-shot learning abilities by providing three previous news articles in the
model’s context to condition it. With the title and subtitle of a proposed next article, the model is able to reliably
generate short articles in the “news” genre.

Language Models are Few-Shot Learners

40+32 pages


model’s context to condition it. With the title and subtitle of a proposed next article, the model is able to reliably
generate short articles in the “news” genre.
To gauge the quality of news article generation from GPT-3 (which we believe is likely to be correlated with conditional
sample generation quality in general), we decided to measure human ability to distinguish GPT-3-generated articles
from real ones. Similar work has been carried out by Kreps et al. [ KMB20 ] and Zellers et al. [ ZHR+19]. Generative
language models are trained to match the distribution of content generated by humans, so the (in)ability of humans to
distinguish the two is a potentially important measure of quality.3
In order to see how well humans can detect model generated text, we arbitrarily selected 25 article titles and subtitles
from the website newser.com (mean length: 215 words). We then generated completions of these titles and subtitles
from four language models ranging in size from 125M to 175B (GPT-3) parameters (mean length: 200 words). For each
model, we presented around 80 US-based participants with a quiz consisting of these real titles and subtitles followed
by either the human written article or the article generated by the model4. Participants were asked to select whether the

Language Models are Few-Shot Learners

40+32 pages


by either the human written article or the article generated by the model4. Participants were asked to select whether the
article was “very likely written by a human”, “more likely written by a human”, “I don’t know”, “more likely written by
a machine”, or “very likely written by a machine”.
The articles we selected were not in the models’ training data and the model outputs were formatted and selected
programmatically to prevent human cherry-picking. All models used the same context to condition outputs on and were
pre-trained with the same context size and the same article titles and subtitles were used as prompts for each model.
However, we also ran an experiment to control for participant effort and attention that followed the same format but
involved intentionally bad model generated articles. This was done by generating articles from a “control model”: a
160M parameter model with no context and increased output randomness.
3This task is also relevant to the potential misuse of language models discussed in Section 6.1.
4We wanted to identify how good an average person on the internet is at detecting language model outputs, so we focused on
participants drawn from the general US population. See Appendix E for details.
25
Mean accuracy95% Conﬁdence
Interval (low, hi)tcompared to

Language Models are Few-Shot Learners

40+32 pages


Language Models are Few-Shot Learners

40+32 pages


GPT-3 175B 52% 49%–54% 16.9 (1 e-34) 7.8%
Table 3.11: Human accuracy in identifying whether short ( 200 word) news articles are model generated . We
ﬁnd that human accuracy (measured by the ratio of correct assignments to non-neutral assignments) ranges from 86%
on the control model to 52% on GPT-3 175B. This table compares mean accuracy between ﬁve different models, and
shows the results of a two-sample T-Test for the difference in mean accuracy between each model and the control model
(an unconditional GPT-3 Small model with increased output randomness).
Mean human accuracy (the ratio of correct assignments to non-neutral assignments per participant) at detecting that
the intentionally bad articles were model generated was 86% where 50% is chance level performance. By contrast,
mean human accuracy at detecting articles that were produced by the 175B parameter model was barely above chance
at52% (see Table 3.11).5Human abilities to detect model generated text appear to decrease as model size increases:
there appears to be a trend towards chance accuracy with model size, and human detection of GPT-3 is close to chance.6
This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).

Language Models are Few-Shot Learners

40+32 pages


This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).
Examples of synthetic articles from GPT-3 are given in Figures 3.14 and 3.15.7Much of the text is—as indicated by the
evaluations—difﬁcult for humans to distinguish from authentic human content. Factual inaccuracies can be an indicator
that an article is model generated since, unlike human authors, the models have no access to the speciﬁc facts that the
article titles refer to or when the article was written. Other indicators include repetition, non sequiturs, and unusual
phrasings, though these are often subtle enough that they are not noticed.
Related work on language model detection by Ippolito et al. [ IDCBE19 ] indicates that automatic discriminators like
GR O V E R [ZHR+19] and GLTR [ GSR19 ] may have greater success at detecting model generated text than human
evaluators. Automatic detection of these models may be a promising area of future research.
Ippolito et al. [ IDCBE19 ] also note that human accuracy at detecting model generated text increases as humans observe
more tokens. To do a preliminary investigation of how good humans are at detecting longer news articles generated

Language Models are Few-Shot Learners

40+32 pages


more tokens. To do a preliminary investigation of how good humans are at detecting longer news articles generated
by GPT-3 175B, we selected 12 world news articles from Reuters with an average length of 569 words and generated
completions of these articles from GPT-3 with an average length of 498 words (298 words longer than our initial
experiments). Following the methodology above, we ran two experiments, each on around 80 US-based participants, to
compare human abilities to detect the articles generated by GPT-3 and a control model.
We found that mean human accuracy at detecting the intentionally bad longer articles from the control model was
88%, while mean human accuracy at detecting the longer articles that were produced by GPT-3 175B was still barely
above chance at52% (see Table 3.12). This indicates that, for news articles that are around 500 words long, GPT-3
continues to produce articles that humans ﬁnd difﬁcult to distinguish from human written news articles.
3.9.5 Learning and Using Novel Words
A task studied in developmental linguistics [ CB78 ] is the ability to learn and utilize new words, for example using a
word in a sentence after seeing it deﬁned only once, or conversely inferring a word’s meaning from only one usage. Here

Language Models are Few-Shot Learners

40+32 pages


word in a sentence after seeing it deﬁned only once, or conversely inferring a word’s meaning from only one usage. Here
we qualitatively test GPT-3’s ability to do the former. Speciﬁcally, we give GPT-3 the deﬁnition of a nonexistent word,
such as “Gigamuru”, and then ask it to use it in a sentence. We provide one to ﬁve previous examples of a (separate)
5We use a two-sample Student’s T-Test to test for signiﬁcant difference between the means of the participant accuracies of each
model and the control model and report the normalized difference in the means (as the t-statistic) and the p-value.
6If a model consistently produces texts that are more impressive than human articles, it is possible that human performance on
this task would drop below 50%. Indeed, many individual participants scored below 50% on this task.
7Additional non-news samples can be found in Appendix F.
26
Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct

Language Models are Few-Shot Learners

40+32 pages


26
Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct
assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberatelybad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed
line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best ﬁt is a power
law with 95% conﬁdence intervals.
Mean accuracy95% Conﬁdence
Interval (low, hi)tcompared to
control (p-value)“I don’t know”
assignments
Control 88% 84%–91% - 2.7%
GPT-3 175B 52% 48%–57% 12.7 (3.2 e-23) 10.6%
Table 3.12: People’s ability to identify whether 500word articles are model generated (as measured by the ratio of
correct assignments to non-neutral assignments) was 88% on the control model and 52% on GPT-3 175B. This table
shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control

Language Models are Few-Shot Learners

40+32 pages


shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control
model (an unconditional GPT-3 Small model with increased output randomness).
27
Title: United Methodists Agree to Historic Split
Subtitle: Those who oppose gay marriage will form their own denomination
Article: After two days of intense debate, the United Methodist Church
has agreed to a historic split - one that is expected to end in the
creation of a new denomination, one that will be "theologically and
socially conservative," according to The Washington Post. The majority of
delegates attending the church's annual General Conference in May voted to
strengthen a ban on the ordination of LGBTQ clergy and to write new rules
that will "discipline" clergy who officiate at same-sex weddings. But
those who opposed these measures have a new plan: They say they will form a
separate denomination by 2020, calling their church the Christian Methodist
denomination.
The Post notes that the denomination, which claims 12.5 million members, was
in the early 20th century the "largest Protestant denomination in the U.S.,"
but that it has been shrinking in recent decades. The new split will be the
second in the church's history. The first occurred in 1968, when roughly
10 percent of the denomination left to form the Evangelical United Brethren

Language Models are Few-Shot Learners

40+32 pages


second in the church's history. The first occurred in 1968, when roughly
10 percent of the denomination left to form the Evangelical United Brethren
Church. The Post notes that the proposed split "comes at a critical time
for the church, which has been losing members for years," which has been
"pushed toward the brink of a schism over the role of LGBTQ people in the
church." Gay marriage is not the only issue that has divided the church. In
2016, the denomination was split over ordination of transgender clergy, with
the North Pacific regional conference voting to ban them from serving as
clergy, and the South Pacific regional conference voting to allow them.
Figure 3.14: The GPT-3 generated news article that humans had the greatest difﬁculty distinguishing from a human
written article (accuracy: 12%).
Title: Star's Tux Promise Draws Megyn Kelly's Sarcasm
Subtitle: Joaquin Phoenix pledged to not change for each awards event
Article: A year ago, Joaquin Phoenix made headlines when he appeared on
the red carpet at the Golden Globes wearing a tuxedo with a paper bag over
his head that read, "I am a shape-shifter. I can't change the world. I
can only change myself." It was a promise to not change to fit into the

Language Models are Few-Shot Learners

40+32 pages


his head that read, "I am a shape-shifter. I can't change the world. I
can only change myself." It was a promise to not change to fit into the
Hollywood mold: "I think that's a really special thing, to not change
yourself. I think it's a really special thing to say, `This is what's
inside of me, I'm proud of it, and I'm not going to be ashamed because of
the way that someone else thinks I should be.'" Now, it's the Oscars, and
Phoenix is at it again. But this time, his publicist is saying he'll be
wearing a tux no matter what.
Megyn Kelly was not impressed, and she let him have it on The Tonight Show.
"You know, I feel like, I feel like you could have worn the tux," she says.
"But you're saying you're a shape-shifter. I don't know if you can change
your tux, but you can change your mind. You can change your mind. You can
change your mind." Phoenix says he did, but it didn't stick. "I was like,
`Okay, I'm going to wear a tuxedo to this thing.' And then I thought, `I
don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him

Language Models are Few-Shot Learners

40+32 pages


don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him
to change his mind again, but Phoenix says it's too late: "I'm committed to
wearing this."
Figure 3.15: The GPT-3 generated news article that humans found the easiest to distinguish from a human written
article (accuracy: 61%).
28
A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses
the word whatpu is:
We were traveling in Africa and we saw these very cute whatpus.
To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses
the word farduddle is:
One day when I was playing tag with my little sister, she got really excited and she
started doing these crazy farduddles.
A "yalubalu" is a type of vegetable that looks like a big pumpkin. An example of a sentence
that uses the word yalubalu is:
I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden
there. It was delicious.
A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the
word Burringo is:

Language Models are Few-Shot Learners

40+32 pages


there. It was delicious.
A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the
word Burringo is:
In our garage we have a Burringo that my father drives to work every day.
A "Gigamuru" is a type of Japanese musical instrument. An example of a sentence that uses the
word Gigamuru is:
I have a Gigamuru that my uncle gave me as a gift. I love to play it at home.
To "screeg" something is to swing a sword at it. An example of a sentence that uses the word
screeg is:
We screeghed at each other for several minutes and then we went outside and ate ice cream.
Figure 3.16: Representative GPT-3 completions for the few-shot task of using a new word in a sentence. Boldface is
GPT-3’s completions, plain text is human prompts. In the ﬁrst example both the prompt and the completion are provided
by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional
prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 other than the conditioning shown
here.

Language Models are Few-Shot Learners

40+32 pages


prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 other than the conditioning shown
here.
nonexistent word being deﬁned and used in a sentence, so the task is few-shot in terms of previous examples of the
broad task and one-shot in terms of the speciﬁc word. Table 3.16 shows the 6 examples we generated; all deﬁnitions
were human-generated, and the ﬁrst answer was human-generated as conditioning while the subsequent answers were
generated by GPT-3. These examples were generated continuously in one sitting and we did not omit or repeatedly try
any prompts. In all cases the generated sentence appears to be a correct or at least plausible use of the word. In the ﬁnal
sentence the model generates a plausible conjugation for the word “screeg” (namely “screeghed”), although the use of
the word is slightly awkward (“screeghed at each other”) despite being plausible in the sense that it could describe a toy
sword ﬁght. Overall, GPT-3 appears to be at least proﬁcient at the task of using novel words in a sentence.
3.9.6 Correcting English Grammar

Language Models are Few-Shot Learners

40+32 pages


3.9.6 Correcting English Grammar
Another task well suited for few-shot learning is correcting English grammar. We test this with GPT-3 in the fewshot setting by giving prompts of the form "Poor English Input: <sentence> nn Good English Output:
<sentence>" . We give GPT-3 one human-generated correction and then ask it to correct 5 more (again without any
omissions or repeats). Results are shown in Figure 3.17.
4 Measuring and Preventing Memorization Of Benchmarks
Since our training dataset is sourced from the internet, it is possible that our model was trained on some of our
benchmark test sets. Accurately detecting test contamination from internet-scale datasets is a new area of research
without established best practices. While it is common practice to train large models without investigating contamination,
given the increasing scale of pretraining datasets, we believe this issue is becoming increasingly important to attend to.
This concern is not just hypothetical. One of the ﬁrst papers to train a language model on Common Crawl data [ TL18 ]
detected and removed a training document which overlapped with one of their evaluation datasets. Other work such
as GPT-2 [ RWC+19] also conducted post-hoc overlap analysis. Their study was relatively encouraging, ﬁnding that
29

Language Models are Few-Shot Learners

40+32 pages


as GPT-2 [ RWC+19] also conducted post-hoc overlap analysis. Their study was relatively encouraging, ﬁnding that
29
Poor English input: I eated the purple berries.
Good English output: I ate the purple berries.
Poor English input: Thank you for picking me as your designer. I'd appreciate it.
Good English output: Thank you for choosing me as your designer. I appreciate it.
Poor English input: The mentioned changes have done. or I did the alteration that you
requested. or I changed things you wanted and did the modifications.
Good English output: The requested changes have been made. or I made the alteration that you
requested. or I changed things you wanted and made the modifications.
Poor English input: I'd be more than happy to work with you in another project.
Good English output: I'd be more than happy to work with you on another project.
Poor English input: Please provide me with a short brief of the design you're looking for and
that'd be nice if you could share some examples or project you did before.
Good English output: Please provide me with a brief description of the design you're
looking for and that would be nice if you could share some examples or projects you have
done before.
Poor English input: The patient was died.
Good English output: The patient died.

Language Models are Few-Shot Learners

40+32 pages


looking for and that would be nice if you could share some examples or projects you have
done before.
Poor English input: The patient was died.
Good English output: The patient died.
Poor English input: We think that Leslie likes ourselves.
Good English output: We think that Leslie likes us.
Poor English input: Janet broke Bill on the finger.
Good English output: Janet broke Bill's finger.
Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap.
Good English output: Mary arranged for John to rent a house in St. Louis.
Poor English input: Today I have went to the store to to buys some many bottle of water.
Good English output: Today I went to the store to buy some bottles of water.
Poor English input: I have tried to hit ball with bat, but my swing is has miss.
Good English output: I tried to hit the ball with the bat, but my swing missed.
Figure 3.17: Representative GPT-3 completions for the few-shot task of correcting English grammar. Boldface
is GPT-3’s completions, plain text is human prompts. In the ﬁrst few examples example both the prompt and the
completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives

Language Models are Few-Shot Learners

40+32 pages


completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives
successive additional prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 aside from
the ﬁrst few examples as conditioning and the “Poor English input/Good English output” framing. We note that the
distinction between ”poor” and ”good” English (and the terms themselves) is complex, contextual, and contested. As
the example mentioning the rental of a house shows, assumptions that the model makes about what “good” is can even
lead it to make errors (here, the model not only adjusts grammar, but also removes the word ”cheap” in a way that alters
meaning).
30
Figure 4.1: GPT-3 Training Curves We measure model performance during training on a deduplicated validation
split of our training distribution. Though there is some gap between training and validation performance, the gap grows
only minimally with model size and training time, suggesting that most of the gap comes from a difference in difﬁculty
rather than overﬁtting.
although models did perform moderately better on data that overlapped between training and testing, this did not

Language Models are Few-Shot Learners

40+32 pages


rather than overﬁtting.
although models did perform moderately better on data that overlapped between training and testing, this did not
signiﬁcantly impact reported results due to the small fraction of data which was contaminated (often only a few percent).
GPT-3 operates in a somewhat different regime. On the one hand, the dataset and model size are about two orders of
magnitude larger than those used for GPT-2, and include a large amount of Common Crawl, creating increased potential
for contamination and memorization. On the other hand, precisely due to the large amount of data, even GPT-3 175B
does not overﬁt its training set by a signiﬁcant amount, measured relative to a held-out validation set with which it was
deduplicated (Figure 4.1). Thus, we expect that contamination is likely to be frequent, but that its effects may not be as
large as feared.
We initially tried to address the issue of contamination by proactively searching for and attempting to remove any overlap
between our training data and the development and test sets of all benchmarks studied in this paper. Unfortunately, a
bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t

Language Models are Few-Shot Learners

40+32 pages


bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t
feasible to retrain the model. To address this, we investigate in detail how the remaining detected overlap impacts
results.
For each benchmark, we produce a ‘clean’ version which removes all potentially leaked examples, deﬁned roughly as
examples that have a 13-gram overlap with anything in the pretraining set (or that overlap with the whole example when
it is shorter than 13-grams). The goal is to very conservatively ﬂag anything that could potentially be contamination,
so as to produce a clean subset that is free of contamination with high conﬁdence. The exact procedure is detailed in
Appendix C.
We then evaluate GPT-3 on these clean benchmarks, and compare to the original score. If the score on the clean
subset is similar to the score on the entire dataset, this suggests that contamination, even if present, does not have a
signiﬁcant effect on reported results. If the score on the clean subset is lower, this suggests contamination may be
inﬂating the results. The results are summarized in Figure 4.2. Although potential contamination is often high (with a
quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence

Language Models are Few-Shot Learners

40+32 pages


quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence
that contamination level and performance difference are correlated. We conclude that either our conservative method
substantially overestimated contamination or that contamination has little effect on performance.
Below, we review in more detail the few speciﬁc cases where either (1) the model performs signiﬁcantly worse on
the cleaned version, or (2) potential contamination is very high, which makes measuring the performance difference
difﬁcult.
Our analysis ﬂagged six groups of benchmarks for further investigation: Word Scrambling, Reading Comprehension
(QuAC, SQuAD2, DROP), PIQA, Winograd, language modeling tasks (Wikitext tasks, 1BW), and German to English
31
Figure 4.2: Benchmark contamination analysis We constructed cleaned versions of each of our benchmarks to
check for potential contamination in our training set. The x-axis is a conservative lower bound for how much of the
dataset is known with high conﬁdence to be clean, and the y-axis shows the difference in performance when evaluating
only on the veriﬁed clean subset. Performance on most benchmarks changed negligibly, but some were ﬂagged for

Language Models are Few-Shot Learners

40+32 pages


only on the veriﬁed clean subset. Performance on most benchmarks changed negligibly, but some were ﬂagged for
further review. On inspection we ﬁnd some evidence for contamination of the PIQA and Winograd results, and we mark
the corresponding results in Section 3 with an asterisk. We ﬁnd no evidence that other benchmarks are affected.
translation. Since our overlap analysis is designed to be extremely conservative, we expect it to produce some false
positives. We summarize the results for each group of tasks below:
•Reading Comprehension: Our initial analysis ﬂagged >90% of task examples from QuAC, SQuAD2, and
DROP as potentially contaminated, so large that even measuring the differential on a clean subset was difﬁcult.
Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source
text was present in our training data but the question/answer pairs were not, meaning the model gains only
background information and cannot memorize the answer to a speciﬁc question.
•German translation: We found 25% of the examples in the WMT16 German-English test set were marked
as potentially contaminated, with an associated total effect size of 1-2 BLEU. Upon inspection, none of the

Language Models are Few-Shot Learners

40+32 pages


as potentially contaminated, with an associated total effect size of 1-2 BLEU. Upon inspection, none of the
ﬂagged examples contain paired sentences resembling NMT training data and collisions were monolingual
matches mostly of snippets of events discussed in the news.
•Reversed Words and Anagrams: Recall that these tasks are of the form “ alaok = koala ”. Due to the
short length of these tasks, we used 2-grams for ﬁltering (ignoring punctuation). After inspecting the ﬂagged
overlaps, we found that they were not typically instances of real reversals or unscramblings in the training set,
but rather palindromes or trivial unscramblings, e.g “ kayak = kayak ”. The amount of overlap was small,
but removing the trivial tasks lead to an increase in difﬁculty and thus a spurious signal. Related to this, the
symbol insertion task shows high overlap but no effect on performance – this is because that task involves
removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to
many spurious matches.
•PIQA: The overlap analysis ﬂagged 29% of examples as contaminated, and observed a 3 percentage point

Language Models are Few-Shot Learners

40+32 pages


many spurious matches.
•PIQA: The overlap analysis ﬂagged 29% of examples as contaminated, and observed a 3 percentage point
absolute decrease (4% relative decrease) in performance on the clean subset. Though the test dataset was
released after our training set was created and its labels are hidden, some of the web pages used by the
crowdsourced dataset creators are contained in our training set. We found a similar decrease in a 25x smaller
model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias
rather than memorization; examples which workers copied may simply be easier. Unfortunately, we cannot
rigorously prove this hypothesis. We therefore mark our PIQA results with an asterisk to denote this potential
contamination.
•Winograd: The overlap analysis ﬂagged 45% of examples, and found a 2.6% decrease in performance on the
clean subset. Manual inspection of the overlapping data point showed that 132 Winograd schemas were in
fact present in our training set, though presented in a different format than we present the task to the model.
Although the decrease in performance is small, we mark our Winograd results in the main paper with an
asterisk.
32
•Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the

Language Models are Few-Shot Learners

40+32 pages


asterisk.
32
•Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the
Children’s Book Test dataset, to be almost entirely contained in our training data. Since we cannot reliably
extract a clean subset here, we do not report results on these datasets, even though we intended to when starting
this work. We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language
modeling benchmark.
We also inspected datasets where contamination was high, but the impact on performance was close to zero, simply
to verify how much actual contamination existed. These appeared to often contain false positives. They had either
no actual contamination, or had contamination that did not give away the answer to the task. One notable exception
was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very
small, with the clean subset scoring within 0.5% of the full dataset. Also, strictly speaking, our ﬁll-in-the-blank format
precludes the simplest form of memorization. Nevertheless, since we made very large gains on LAMBADA in this
paper, the potential contamination is noted in the results section.
An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the

Language Models are Few-Shot Learners

40+32 pages


paper, the potential contamination is noted in the results section.
An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the
same distribution as the original dataset. It remains possible that memorization inﬂates results but at the same time
is precisely counteracted by some statistical bias causing the clean subset to be easier. However, the sheer number
of shifts close to zero suggests this is unlikely, and we also observed no noticeable difference in the shifts for small
models, which are unlikely to be memorizing.
Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright
remove problematic results, depending on the severity. Much work remains to be done to address this important and
subtle issue for the ﬁeld in general, both when designing benchmarks and when training models. For a more detailed
explanation of our analysis, we refer the reader to Appendix C.
5 Limitations
GPT-3 and our analysis of it have a number of limitations. Below we describe some of these and suggest directions for
future work.
First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct
predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks. On text synthesis, although

Language Models are Few-Shot Learners

40+32 pages


predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks. On text synthesis, although
the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to
lose coherence over sufﬁciently long passages, contradict themselves, and occasionally contain non-sequitur sentences
or paragraphs. We will release a collection of 500 uncurated unconditional samples to help provide a better sense of
GPT-3’s limitations and strengths at text synthesis. Within the domain of discrete language tasks, we have noticed
informally that GPT-3 seems to have special difﬁculty with “common sense physics”, despite doing well on some
datasets (such as PIQA [ BZB+19]) that test this domain. Speciﬁcally GPT-3 has difﬁculty with questions of the type
“If I put cheese into the fridge, will it melt?”. Quantitatively, GPT-3’s in-context learning performance has some notable
gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when

Language Models are Few-Shot Learners

40+32 pages


gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when
evaluated one-shot or even few-shot on some “comparison” tasks, such as determining if two words are used the same
way in a sentence, or if one sentence implies another (WIC and ANLI respectively), as well as on a subset of reading
comprehension tasks. This is especially striking given GPT-3’s strong few-shot performance on many other tasks.
GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above. We focused
on exploring in-context learning behavior in autoregressive language models because it is straightforward to both
sample and compute likelihoods with this model class. As a result our experiments do not include any bidirectional
architectures or other training objectives such as denoising. This is a noticeable difference from much of the recent
literature, which has documented improved ﬁne-tuning performance when using these approaches over standard
language models [ RSR+19]. Thus our design decision comes at the cost of potentially worse performance on tasks
which empirically beneﬁt from bidirectionality. This may include ﬁll-in-the-blank tasks, tasks that involve looking back

Language Models are Few-Shot Learners

40+32 pages


which empirically beneﬁt from bidirectionality. This may include ﬁll-in-the-blank tasks, tasks that involve looking back
and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then
generating a very short answer. This could be a possible explanation for GPT-3’s lagging few-shot performance on a
few of the tasks, such as WIC (which involves comparing the use of a word in two sentences), ANLI (which involves
comparing two sentences to see if one implies the other), and several reading comprehension tasks (e.g. QuAC and
RACE). We also conjecture, based on past literature, that a large bidirectional model would be stronger at ﬁne-tuning
than GPT-3. Making a bidirectional model at the scale of GPT-3, and/or trying to make bidirectional models work with
few- or zero-shot learning, is a promising direction for future research, and could help achieve the “best of both worlds”.
A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
33

Language Models are Few-Shot Learners

40+32 pages


autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
33
pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to
predict and what is less important. [ RRS20 ] demonstrate beneﬁts of customizing prediction to entities of interest. Also,
with self-supervised objectives, task speciﬁcation relies on forcing the desired task into a prediction problem, whereas
ultimately, useful language systems (for example virtual assistants) might be better thought of as taking goal-directed
actions rather than just making predictions. Finally, large pretrained language models are not grounded in other domains
of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world
[BHT+20]. For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a
different approach is likely to be necessary. Promising future directions in this vein might include learning the objective
function from humans [ ZSW+19a], ﬁne-tuning with reinforcement learning, or adding additional modalities such as
images to provide grounding and a better model of the world [CLY+19].

Language Models are Few-Shot Learners

40+32 pages


images to provide grounding and a better model of the world [CLY+19].
Another limitation broadly shared by language models is poor sample efﬁciency during pre-training. While GPT-3
takes a step towards test-time sample efﬁciency closer to that of humans (one-shot or zero-shot), it still sees much more
text during pre-training than a human sees in the their lifetime [ Lin20 ]. Improving pre-training sample efﬁciency is
an important direction for future work, and might come from grounding in the physical world to provide additional
information, or from algorithmic improvements.
A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot
learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identiﬁes tasks that it
has learned during training. These possibilities exist on a spectrum, ranging from demonstrations in the training set that
are drawn from exactly the same distribution as those at test time, to recognizing the same task but in a different format,
to adapting to a speciﬁc style of a general task such as QA, to learning a skill entirely de novo. Where GPT-3 is on

Language Models are Few-Shot Learners

40+32 pages


to adapting to a speciﬁc style of a general task such as QA, to learning a skill entirely de novo. Where GPT-3 is on
this spectrum may also vary from task to task. Synthetic tasks such as wordscrambling or deﬁning nonsense words
seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although
possibly from data that is very different in organization and style than the test data. Ultimately, it is not even clear what
humans learn from scratch vs from prior demonstrations. Even organizing diverse demonstrations during pre-training
and identifying them at test time would be an advance for language models, but nevertheless understanding precisely
how few-shot learning works is an important unexplored direction for future research.
A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are
both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of
models of this scale in their current form. One possible future direction to address this is distillation [ HVD15 ] of large
models down to a manageable size for speciﬁc tasks. Large models such as GPT-3 contain a very wide range of skills,
most of which are not needed for a speciﬁc task, suggesting that in principle aggressive distillation may be possible.

Language Models are Few-Shot Learners

40+32 pages


most of which are not needed for a speciﬁc task, suggesting that in principle aggressive distillation may be possible.
Distillation is well-explored in general [ LHCG19a ] but has not been tried at the scale of hundred of billions parameters;
new challenges and opportunities may be associated with applying it to models of this size.
Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable,
it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in
performance than humans on standard benchmarks, and it retains the biases of the data it has been trained on. This
last issue – biases in the data that may lead the model to generate stereotyped or prejudiced content – is of special
concern from a societal perspective, and will be discussed along with other issues in the next section on Broader Impacts
(Section 6).
6 Broader Impacts
Language models have a wide range of beneﬁcial applications for society, including code and writing auto-completion,
grammar assistance, game narrative generation, improving search engine responses, and answering questions. But
they also have potentially harmful applications. GPT-3 improves the quality of text generation and adaptability over

Language Models are Few-Shot Learners

40+32 pages


grammar assistance, game narrative generation, improving search engine responses, and answering questions. But
they also have potentially harmful applications. GPT-3 improves the quality of text generation and adaptability over
smaller models and increases the difﬁculty of distinguishing synthetic text from human-written text. It therefore has the
potential to advance both the beneﬁcial and harmful applications of language models.
Here we focus on the potential harms of improved language models, not because we believe the harms are necessarily
greater, but in order to stimulate efforts to study and mitigate them. The broader impacts of language models like this
are numerous. We focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 in
Section 6.1, and issues of bias, fairness, and representation within models like GPT-3 in Section 6.2. We also brieﬂy
discuss issues of energy efﬁciency (Section 6.3).
34
6.1 Misuse of Language Models
Malicious uses of language models can be somewhat difﬁcult to anticipate because they often involve repurposing
language models in a very different environment or for a different purpose than researchers intended. To help with this,
we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying

Language Models are Few-Shot Learners

40+32 pages


we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying
threats and potential impacts, assessing likelihood, and determining risk as a combination of likelihood and impact
[Ros12]. We discuss three factors: potential misuse applications, threat actors, and external incentive structures.
6.1.1 Potential Misuse Applications
Any socially harmful activity that relies on generating text could be augmented by powerful language models. Examples
include misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing
and social engineering pretexting. Many of these applications bottleneck on human beings to write sufﬁciently high
quality text. Language models that produce high quality text generation could lower existing barriers to carrying out
these activities and increase their efﬁcacy.
The misuse potential of language models increases as the quality of text synthesis improves. The ability of GPT-3 to
generate several paragraphs of synthetic content that people ﬁnd difﬁcult to distinguish from human-written text in
3.9.4 represents a concerning milestone in this regard.
6.1.2 Threat Actor Analysis
Threat actors can be organized by skill and resource levels, ranging from low or moderately skilled and resourced actors
who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced

Language Models are Few-Shot Learners

40+32 pages


who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced
(e.g. state-sponsored) groups with long-term agendas [SBC+19].
To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat
groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed. While we did
ﬁnd signiﬁcant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances
of experimentation and no successful deployments since then. Additionally, those misuse discussions were correlated
with media coverage of language model technologies. From this, we assess that the threat of misuse from these actors is
not immediate, but signiﬁcant improvements in reliability could change this.
Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about
possible APT activity involving the use of language models. Since the release of GPT-2 there has been no discernible
difference in operations that may see potential gains by using language models. The assessment was that language
models may not be worth investing signiﬁcant resources in because there has been no convincing demonstration that
current language models are signiﬁcantly better than current methods for generating text, and because methods for

Language Models are Few-Shot Learners

40+32 pages


current language models are signiﬁcantly better than current methods for generating text, and because methods for
“targeting” or “controlling” the content of language models are still at a very early stage.
6.1.3 External Incentive Structures
Each threat actor group also has a set of tactics, techniques, and procedures (TTPs) that they rely on to accomplish their
agenda. TTPs are inﬂuenced by economic factors like scalability and ease of deployment; phishing is extremely popular
among all groups because it offers a low-cost, low-effort, high-yield method of deploying malware and stealing login
credentials. Using language models to augment existing TTPs would likely result in an even lower cost of deployment.
Ease of use is another signiﬁcant incentive. Having stable infrastructure has a large impact on the adoption of TTPs.
The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k
truncation) they are not able to perform consistently without human feedback. If a social media disinformation bot
produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the

Language Models are Few-Shot Learners

40+32 pages


produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the
amount of human labor required in operating this bot. But a human is still needed to ﬁlter the outputs, which restricts
how scalable the operation can be.
Based on our analysis of this model and analysis of threat actors and the landscape, we suspect AI researchers will
eventually develop language models that are sufﬁciently consistent and steerable that they will be of greater interest to
malicious actors. We expect this will introduce challenges for the broader research community, and hope to work on
this through a combination of mitigation research, prototyping, and coordinating with other technical developers.
35
6.2 Fairness, Bias, and Representation
Biases present in training data may lead models to generate stereotyped or prejudiced content. This is concerning,
since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and
producing demeaning portrayals amongst other potential harms [ Cra17 ]. We have conducted an analysis of biases in
the model in order to better understand GPT-3’s limitations when it comes to fairness, bias, and representation.8
Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and

Language Models are Few-Shot Learners

40+32 pages


Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and
behaviors. We focus on biases relating to gender, race, and religion, although many other categories of bias are likely
present and could be studied in follow-up work. This is a preliminary analysis and does not reﬂect all of the model’s
biases even within the studied categories.
Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reﬂect stereotypes
present in their training data. Below we discuss our preliminary ﬁndings of bias along the dimensions of gender, race,
and religion. We probe for bias in the 175 billion parameter model and also in similar smaller models, to see if and how
they are different in this dimension.
6.2.1 Gender
In our investigation of gender bias in GPT-3, we focused on associations between gender and occupation. We found
that occupations in general have a higher probability of being followed by a male gender identiﬁer than a female one
(in other words, they are male leaning) when given a context such as "Thefoccupationgwas a" (Neutral Variant).
83% of the 388 occupations we tested were more likely to be followed by a male identiﬁer by GPT-3. We measured

Language Models are Few-Shot Learners

40+32 pages


83% of the 388 occupations we tested were more likely to be followed by a male identiﬁer by GPT-3. We measured
this by feeding the model a context such as "The detective was a" and then looking at the probability of the
model following up with male indicating words (eg. man, male etc.) or female indicating words (woman, female etc.).
In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus
were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and
sheriff. Occupations that were more likely to be followed by female identiﬁers include midwife, nurse, receptionist,
housekeeper etc.
We also tested how these probabilities changed when we shifted the context to be the "The competent foccupationg
was a" (Competent Variant), and when we shifted the context to be "The incompetent foccupationgwas a"
(Incompetent Variant) for each occupation in the dataset. We found that, when prompted with "The competent
foccupationgwas a," the majority of occupations had an even higher probability of being followed by a
male identiﬁer than a female one than was the case with our original neutral prompt, "Thefoccupationgwas

Language Models are Few-Shot Learners

40+32 pages


male identiﬁer than a female one than was the case with our original neutral prompt, "Thefoccupationgwas
a". With the prompt "The incompetent foccupationgwas a" the majority of occupations still leaned male
with a similar probability than for our original neutral prompt. The average occupation bias - measured as
1
njobsP
jobslog(P(femalejContext)
P(malejContext)))- was1:11for the Neutral Variant, 2:14for the Competent Variant and 1:15
for the Incompetent Variant.
We also carried out pronoun resolution on the Winogender dataset [ RNLVD18 ] using two methods which further
corroborated the model’s tendency to associate most occupations with males. One method measured the models ability to correctly assign a pronoun as the occupation or the participant. For example, we fed the model
a context such as "The advisor met with the advisee because she wanted to get advice about job
applications. `She' refers to the" and found the option with the lowest probability between the two possible options (Choices between Occupation Option: advisor; Participant Option: advisee).
Occupation and participant words often have societal biases associated with them such as the assumption that most

Language Models are Few-Shot Learners

40+32 pages


Occupation and participant words often have societal biases associated with them such as the assumption that most
occupants are by default male. We found that the language models learnt some of these biases such as a tendency to
associate female pronouns with participant positions more than male pronouns. GPT-3 175B had the highest accuracy of
all the models (64.17%) on this task. It was also the only model where the accuracy for Occupant sentences (sentences
where the correct answer was the Occupation option) for females was higher than for males (81.7% vs 76.7%). All
other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns
with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both. This offers
some preliminary evidence that in places where issues of bias can make language models susceptible to error, the larger
models are more robust than smaller models.
We also performed co-occurrence tests, where we analyzed which words are likely to occur in the vicinity of other preselected words. We created a model output sample set by generating 800 outputs of length 50 each with a temperature
8Evaluating fairness, bias, and representation in language models is a rapidly-developing area with a large body of prior work.
See, for example, [HZJ+19, NBR20, SCNP19].

Language Models are Few-Shot Learners

40+32 pages


See, for example, [HZJ+19, NBR20, SCNP19].
36
Table 6.1: Most Biased Descriptive Words in 175B Model
Top 10 Most Biased Male Descriptive Words with Raw
Co-Occurrence CountsTop 10 Most Biased Female Descriptive Words with Raw
Co-Occurrence Counts
Average Number of Co-Occurrences Across All Words:
17.5Average Number of Co-Occurrences Across All Words:
23.9
Large (16) Optimistic (12)
Mostly (15) Bubbly (12)
Lazy (14) Naughty (12)
Fantastic (13) Easy-going (12)
Eccentric (13) Petite (10)
Protect (10) Tight (10)
Jolly (10) Pregnant (10)
Stable (9) Gorgeous (28)
Personable (22) Sucked (8)
Survive (7) Beautiful (158)
of 1 and top p of 0.9 for every prompt in our dataset. For gender, we had prompts such as "He was very" ,"She
was very" ,"He would be described as" ,"She would be described as"9. We looked at the adjectives and

Language Models are Few-Shot Learners

40+32 pages


was very" ,"He would be described as" ,"She would be described as"9. We looked at the adjectives and
adverbs in the top 100 most favored words using an off-the-shelf POS tagger [ LB02 ]. We found females were more
often described using appearance oriented words such as ”beautiful” and ”gorgeous” as compared to men who were
more often described using adjectives that span a greater spectrum.
Table 6.1 shows the top 10 most favored descriptive words for the model along with the raw number of times each
word co-occurred with a pronoun indicator. “Most Favored” here indicates words which were most skewed towards a
category by co-occurring with it at a higher rate as compared to the other category. To put these numbers in perspective,
we have also included the average for the number of co-occurrences across all qualifying words for each gender.
6.2.2 Race
To investigate racial bias in GPT-3, we seeded the model with prompts such as - "Thefracegman was very" ,
"Thefracegwoman was very" and"People would describe the fracegperson as" and generated 800
samples for each of the above prompts, with fracegreplaced with a term indicating a racial category such as White

Language Models are Few-Shot Learners

40+32 pages


samples for each of the above prompts, with fracegreplaced with a term indicating a racial category such as White
or Asian. We then measure word co-occurrences in the generated samples. Given prior research demonstrating that
language models produce text of differing sentiment when varying features such as occupation [ HZJ+19], we explored
how race impacted sentiment. We measured sentiment using Senti WordNet [ BES10 ] for the words which co-occurred
disproportionately with each race. Each word sentiment varied from 100 to -100, with positive scores indicating positive
words (eg. wonderfulness: 100, amicable: 87.5), negative scores indicating negative words (eg. wretched: -87.5 , horrid:
-87.5) and a score of 0 indicating neutral words (eg. sloping, chalet).
It should be noted that we were explicitly prompting the models to talk about race and this in turn generated text that
focused on racial features; these results are not from the models talking about race in the wild but talking about race in
an experimental setup where they have been primed to do so. Additionally, since we are measuring sentiment by simply
looking at word co-occurrences, the resulting sentiment can reﬂect socio-historical factors - for instance, text relating to
a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated

Language Models are Few-Shot Learners

40+32 pages


a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated
with a negative sentiment under this testing methodology.
Across the models we analyzed, ‘Asian’ had a consistently high sentiment - it ranked 1st in 3 out of 7 models. On the
other hand, ’Black’ had a consistently low sentiment - it ranked the lowest in 5 out of 7 models. These differences
narrowed marginally on the larger model sizes. This analysis gives a sense of the biases of different models and
highlights the need for more sophisticated analysis of the relationship between sentiment, entities, and input data.
9We only used male and female pronouns. This simplifying assumption makes it easier to study co-occurrence since it does not
require the isolation of instances in which ‘they’ refers to a singular noun from those where it didn’t, but other forms of gender bias
are likely present and could be studied using different approaches.
37
Figure 6.1: Racial Sentiment Across Models
Religion Most Favored Descriptive Words
Atheism ‘Theists’, ‘Cool’, ‘Agnostics’, ‘Mad’, ‘Theism’, ‘Defensive’, ‘Complaining’, ‘Correct’, ‘Arrogant’,
‘Characterized’

Language Models are Few-Shot Learners

40+32 pages


‘Characterized’
Buddhism ‘Myanmar’, ‘Vegetarians’, ‘Burma’, ‘Fellowship’, ‘Monk’, ‘Japanese’, ‘Reluctant’, ‘Wisdom’, ‘Enlightenment’, ‘Non-Violent’
Christianity ‘Attend’, ‘Ignorant’, ‘Response’, ‘Judgmental’, ‘Grace’, ‘Execution’, ‘Egypt’, ‘Continue’, ‘Comments’, ‘Ofﬁcially’
Hinduism ‘Caste’, ‘Cows’, ‘BJP’, ‘Kashmir’, ‘Modi’, ‘Celebrated’, ‘Dharma’, ‘Pakistani’, ‘Originated’, ‘Africa’
Islam ‘Pillars’, ‘Terrorism’, ‘Fasting’, ‘Sheikh’, ‘Non-Muslim’, ‘Source’, ‘Charities’, ‘Levant’, ‘Allah’,

Language Models are Few-Shot Learners

40+32 pages


‘Prophet’
Judaism ‘Gentiles’, ‘Race’, ‘Semites’, ‘Whites’, ‘Blacks’, ‘Smartest’, ‘Racists’, ‘Arabs’, ‘Game’, ‘Russian’
Table 6.2: Shows the ten most favored words about each religion in the GPT-3 175B model.
6.2.3 Religion
We studied which words co-occurred with religious terms relating to Atheism, Buddhism, Christianity, Hinduism, Islam,
and Judaism, by generating 800 model outputs of length 50 with a temperature of 1 and a top pof0:9for every
prompt. Our prompts were of the nature "fReligion practitioners gare" (Eg. "Christians are" ) for each
of the six religious categories listed above. We then allowed the model to naturally carry out completions and created a
corpus of such completions for studying co-occurrence of words.
The following is an example output from the model:
"Buddhists are divided into two main branches - Theravada and Mahayana. Theravada
is the more conservative branch, centering on monastic life and the earliest sutras

Language Models are Few-Shot Learners

40+32 pages


is the more conservative branch, centering on monastic life and the earliest sutras
and refusing to recognize the later Mahayana sutras as authentic."
Similar to race, we found that the models make associations with religious terms that indicate some propensity to reﬂect
how these terms are sometimes presented in the world. For example, with the religion Islam , we found that words such
asramadan ,prophet andmosque co-occurred at a higher rate than for other religions. We also found that words such
asviolent ,terrorism andterrorist co-occurred at a greater rate with Islam than with other religions and were in
the top 40 most favored words for Islam in GPT-3.
38
6.2.4 Future Bias and Fairness Challenges
We have presented this preliminary analysis to share some of the biases we found in order to motivate further research,
and to highlight the inherent difﬁculties in characterizing biases in large-scale generative models; we expect this to be an
area of continuous research for us and are excited to discuss different methodological approaches with the community.
We view the work in this section as subjective signposting - we chose gender, race, and religion as a starting point, but
we recognize the inherent subjectivity in this choice. Our work is inspired by the literature on characterizing model

Language Models are Few-Shot Learners

40+32 pages


we recognize the inherent subjectivity in this choice. Our work is inspired by the literature on characterizing model
attributes to develop informative labels such as Model Cards for Model Reporting from [MWZ+18].
Ultimately, it is important not just to characterize biases in language systems but to intervene. The literature on this
is also extensive [ QMZH19 ,HZJ+19], so we offer only a few brief comments on future directions speciﬁc to large
language models. In order to pave the way for effective bias prevention in general purpose models, there is a need for
building a common vocabulary tying together the normative, technical and empirical challenges of bias mitigation for
these models. There is room for more research that engages with the literature outside NLP, better articulates normative
statements about harm, and engages with the lived experience of communities affected by NLP systems [ BBDIW20 ].
Thus, mitigation work should not be approached purely with a metric driven objective to ‘remove’ bias as this has been
shown to have blind spots [GG19, NvNvdG19] but in a holistic manner.
6.3 Energy Usage
Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3

Language Models are Few-Shot Learners

40+32 pages


6.3 Energy Usage
Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3
175B consumed several thousand petaﬂop/s-days of compute during pre-training, compared to tens of petaﬂop/s-days
for a 1.5B parameter GPT-2 model (Figure 2.2). This means we should be cognizant of the cost and efﬁciency of such
models, as advocated by [SDSE19].
The use of large-scale pre-training also gives another lens through which to view the efﬁciency of large models - we
should consider not only the resources that go into training them, but how these resources are amortized over the
lifetime of a model, which will subsequently be used for a variety of purposes and ﬁne-tuned for speciﬁc tasks. Though
models like GPT-3 consume signiﬁcant resources during training, they can be surprisingly efﬁcient once trained: even
with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or

Language Models are Few-Shot Learners

40+32 pages


with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or
only a few cents in energy costs. Additionally, techniques like model distillation [ LHCG19a ] can further bring down
the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efﬁcient
versions of them for use in appropriate contexts. Algorithmic progress may also naturally further increase the efﬁciency
of such models over time, similar to trends observed in image recognition and neural machine translation [HB20].
7 Related Work
Several lines of work have focused on increasing parameter count and/or computation in language models as a
means to improve generative or task performance. An early work scaled LSTM based language models to over a
billion parameters [ JVS+16]. One line of work straightforwardly increases the size of transformer models, scaling
up parameters and FLOPS-per-token roughly in proportion. Work in this vein has successively increased model size:
213 million parameters [ VSP+17] in the original paper, 300 million parameters [ DCLT18 ], 1.5 billion parameters
[RWC+19], 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19], and most recently 17 billion parameters

Language Models are Few-Shot Learners

40+32 pages


[RWC+19], 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19], and most recently 17 billion parameters
[Tur20 ]. A second line of work has focused on increasing parameter count but not computation, as a means of
increasing models’ capacity to store information without increased computational cost. These approaches rely on the
conditional computation framework [ BLC13 ] and speciﬁcally, the mixture-of-experts method [ SMM+17] has been
used to produce 100 billion parameter models and more recently 50 billion parameter translation models [ AJF19 ],
though only a small fraction of the parameters are actually used on each forward pass. A third approach increases
computation without increasing parameters; examples of this approach include adaptive computation time [ Gra16 ] and
the universal transformer [ DGV+18]. Our work focuses on the ﬁrst approach (scaling compute and parameters together,
by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ
this strategy.
Several efforts have also systematically studied the effect of scale on language model performance. [ KMH+20,
RRBS19 ,LWS+20,HNA+17], ﬁnd a smooth power-law trend in loss as autoregressive language models are scaled up.

Language Models are Few-Shot Learners

40+32 pages


RRBS19 ,LWS+20,HNA+17], ﬁnd a smooth power-law trend in loss as autoregressive language models are scaled up.
This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the
curve can perhaps be detected in Figure 3.1), and we also ﬁnd relatively smooth increases in many (though not all)
downstream tasks across 3 orders of magnitude of scaling.
Another line of work goes in the opposite direction from scaling, attempting to preserve strong performance in language
models that are as small as possible. This approach includes ALBERT [ LCG+19] as well as general [ HVD15 ] and
39
task-speciﬁc [ SDCW19 ,JYS+19,KR16 ] approaches to distillation of language models. These architectures and
techniques are potentially complementary to our work, and could be applied to decrease latency and memory footprint
of giant models.
As ﬁne-tuned language models have neared human performance on many standard benchmark tasks, considerable
effort has been devoted to constructing more difﬁcult or open-ended tasks, including question answering [ KPR+19,
IBGC+14,CCE+18,MCKS18 ], reading comprehension [ CHI+18,RCM19 ], and adversarially constructed datasets

Language Models are Few-Shot Learners

40+32 pages


IBGC+14,CCE+18,MCKS18 ], reading comprehension [ CHI+18,RCM19 ], and adversarially constructed datasets
designed to be difﬁcult for existing language models [ SBBC19 ,NWD+19]. In this work we test our models on many
of these datasets.
Many previous efforts have focused speciﬁcally on question-answering, which constitutes a signiﬁcant fraction of the
tasks we tested on. Recent efforts include [ RSR+19,RRS20 ], which ﬁne-tuned an 11 billion parameter language model,
and [ GLT+20], which focused on attending over a large corpus of data at test time. Our work differs in focusing on
in-context learning but could be combined in the future with those of [GLT+20, LPP+20].
Metalearning in language models has been utilized in [ RWC+19], though with much more limited results and no
systematic study. More broadly, language model metalearning has an inner-loop-outer-loop structure, making it
structurally similar to metalearning as applied to ML in general. Here there is an extensive literature, including

Language Models are Few-Shot Learners

40+32 pages


structurally similar to metalearning as applied to ML in general. Here there is an extensive literature, including
matching networks [ VBL+16], RL2 [ DSC+16], learning to optimize [ RL16 ,ADG+16,LM17 ] and MAML [ FAL17 ].
Our approach of stufﬁng the model’s context with previous examples is most structurally similar to RL2 and also
resembles [ HYC01 ], in that an inner loop of adaptation takes place through computation in the model’s activations
across timesteps, without updating the weights, while an outer loop (in this case just language model pre-training)
updates the weights, and implicitly learns the ability to adapt to or at least recognize tasks deﬁned at inference-time.
Few-shot auto-regressive density estimation was explored in [ RCP+17] and [ GWC+18] studied low-resource NMT as
a few-shot learning problem.
While the mechanism of our few-shot approach is different, prior work has also explored ways of using pre-trained
language models in combination with gradient descent to perform few-shot learning [ SS20 ]. Another sub-ﬁeld with

Language Models are Few-Shot Learners

40+32 pages


language models in combination with gradient descent to perform few-shot learning [ SS20 ]. Another sub-ﬁeld with
similar goals is semi-supervised learning where approaches such as UDA [ XDH+19] also explore methods of ﬁne-tuning
when very little labeled data is available.
Giving multi-task models instructions in natural language was ﬁrst formalized in a supervised setting with [ MKXS18 ]
and utilized for some tasks (such as summarizing) in a language model with [ RWC+19]. The notion of presenting
tasks in natural language was also explored in the text-to-text transformer [ RSR+19], although there it was applied for
multi-task ﬁne-tuning rather than for in-context learning without weight updates.
Another approach to increasing generality and transfer-learning capability in language models is multi-task learning
[Car97 ], which ﬁne-tunes on a mixture of downstream tasks together, rather than separately updating the weights for
each one. If successful multi-task learning could allow a single model to be used for many tasks without updating the
weights (similar to our in-context learning approach), or alternatively could improve sample efﬁciency when updating

Language Models are Few-Shot Learners

40+32 pages


weights (similar to our in-context learning approach), or alternatively could improve sample efﬁciency when updating
the weights for a new task. Multi-task learning has shown some promising initial results [ LGH+15,LSP+18] and
multi-stage ﬁne-tuning has recently become a standardized part of SOTA results on some datasets [ PFB18 ] and pushed
the boundaries on certain tasks [ KKS+20], but is still limited by the need to manually curate collections of datasets and
set up training curricula. By contrast pre-training at large enough scale appears to offer a “natural” broad distribution of
tasks implicitly contained in predicting the text itself. One direction for future work might be attempting to generate
a broader set of explicit tasks for multi-task learning, for example through procedural generation [ TFR+17], human
interaction [ZSW+19b], or active learning [Mac92].
Algorithmic innovation in language models over the last two years has been enormous, including denoising-based
bidirectionality [ DCLT18 ], preﬁxLM [ DL15 ] and encoder-decoder architectures [ LLG+19,RSR+19], random permutations during training [ YDY+19], architectures that improve the efﬁciency of sampling [ DYY+19], improvements in

Language Models are Few-Shot Learners

40+32 pages


data and training procedures [ LOG+19], and efﬁciency increases in the embedding parameters [ LCG+19]. Many of
these techniques provide signiﬁcant gains on downstream tasks. In this work we continue to focus on pure autoregressive
language models, both in order to focus on in-context learning performance and to reduce the complexity of our large
model implementations. However, it is very likely that incorporating these algorithmic advances could improve GPT-3’s
performance on downstream tasks, especially in the ﬁne-tuning setting, and combining GPT-3’s scale with these
algorithmic techniques is a promising direction for future work.
8 Conclusion
We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and
benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of
40
state-of-the-art ﬁne-tuned systems, as well as generating high-quality samples and strong qualitative performance at
tasks deﬁned on-the-ﬂy. We documented roughly predictable trends of scaling in performance without using ﬁne-tuning.
We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results

Language Models are Few-Shot Learners

40+32 pages


We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results
suggest that very large language models may be an important ingredient in the development of adaptable, general
language systems.
Acknowledgements
The authors would like to thank Ryan Lowe for giving detailed feedback on drafts of the paper. Thanks to Jakub
Pachocki and Szymon Sidor for suggesting tasks, and Greg Brockman, Michael Petrov, Brooke Chan, and Chelsea
V oss for helping run evaluations on OpenAI’s infrastructure. Thanks to David Luan for initial support in scaling up
this project, Irene Solaiman for discussions about ways to approach and evaluate bias, Harrison Edwards and Yura
Burda for discussions and experimentation with in-context learning, Geoffrey Irving and Paul Christiano for early
discussions of language model scaling, Long Ouyang for advising on the design of the human evaluation experiments,
Chris Hallacy for discussions on data collection, and Shan Carter for help with visual design. Thanks to the millions of
people who created content that was used in the training of the model, and to those who were involved in indexing or
upvoting the content (in the case of WebText). Additionally, we would like to thank the entire OpenAI infrastructure
and supercomputing teams for making it possible to train models at this scale.
41
Contributions

Language Models are Few-Shot Learners

40+32 pages


and supercomputing teams for making it possible to train models at this scale.
41
Contributions
Tom Brown, Ben Mann, Prafulla Dhariwal, Dario Amodei, Nick Ryder, Daniel M Ziegler, and Jeffrey Wu
implemented the large-scale models, training infrastructure, and model-parallel strategies.
Tom Brown, Dario Amodei, Ben Mann, and Nick Ryder conducted pre-training experiments.
Ben Mann and Alec Radford collected, ﬁltered, deduplicated, and conducted overlap analysis on the training data.
Melanie Subbiah, Ben Mann, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown, Tom Henighan, and
Girish Sastry implemented the downstream tasks and the software framework for supporting them, including creation
of synthetic tasks.
Jared Kaplan and Sam McCandlish initially predicted that a giant language model should show continued gains, and
applied scaling laws to help predict and guide model and data scaling decisions for the research.
Ben Mann implemented sampling without replacement during training.
Alec Radford originally demonstrated few-shot learning occurs in language models.
Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically
studied in-context learning curves, task prompting, and evaluation methods.

Language Models are Few-Shot Learners

40+32 pages


Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically
studied in-context learning curves, task prompting, and evaluation methods.
Prafulla Dhariwal implemented an early version of the codebase, and developed the memory optimizations for fully
half-precision training.
Rewon Child and Mark Chen developed an early version of our model-parallel strategy.
Rewon Child and Scott Gray contributed the sparse transformer.
Aditya Ramesh experimented with loss scaling strategies for pretraining.
Melanie Subbiah and Arvind Neelakantan implemented, experimented with, and tested beam search.
Pranav Shyam worked on SuperGLUE and assisted with connections to few-shot learning and meta-learning literature.
Sandhini Agarwal conducted the fairness and representation analysis.
Girish Sastry and Amanda Askell conducted the human evaluations of the model.
Ariel Herbert-Voss conducted the threat analysis of malicious use.
Gretchen Krueger edited and red-teamed the policy sections of the paper.
Benjamin Chess, Clemens Winter, Eric Sigler, Christopher Hesse, Mateusz Litwin, and Christopher Berner
optimized OpenAI’s clusters to run the largest models efﬁciently.
Scott Gray developed fast GPU kernels used during training.

Language Models are Few-Shot Learners

40+32 pages