optimized OpenAI’s clusters to run the largest models efﬁciently.
Scott Gray developed fast GPU kernels used during training.
Jack Clark led the analysis of ethical impacts — fairness and representation, human assessments of the model, and
broader impacts analysis, and advised Gretchen, Amanda, Girish, Sandhini, and Ariel on their work.
Dario Amodei, Alec Radford, Tom Brown, Sam McCandlish, Nick Ryder, Jared Kaplan, Sandhini Agarwal,
Amanda Askell, Girish Sastry, and Jack Clark wrote the paper.
Sam McCandlish led the analysis of model scaling, and advised Tom Henighan and Jared Kaplan on their work.
Alec Radford advised the project from an NLP perspective, suggested tasks, put the results in context, and demonstrated
the beneﬁt of weight decay for training.
Ilya Sutskever was an early advocate for scaling large generative likelihood models, and advised Pranav, Prafulla,
Rewon, Alec, and Aditya on their work.
Dario Amodei designed and led the research.
42
A Details of Common Crawl Filtering
As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1)

Language Models are Few-Shot Learners

40+32 pages


42
A Details of Common Crawl Filtering
As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1)
ﬁltering Common Crawl and (2) fuzzy deduplication:
1.In order to improve the quality of Common Crawl, we developed an automatic ﬁltering method to remove low
quality documents. Using the original WebText as a proxy for high-quality documents, we trained a classiﬁer
to distinguish these from raw Common Crawl. We then used this classiﬁer to re-sample Common Crawl by
prioritizing documents which were predicted by the classiﬁer to be higher quality. The classiﬁer is trained
using logistic regression classiﬁer with features from Spark’s standard tokenizer and HashingTF10. For the
positive examples, we used a collection of curated datasets such as WebText, Wikiedia, and our web books
corpus as the positive examples, and for the negative examples, we used unﬁltered Common Crawl. We used
this classiﬁer to score Common Crawl documents. We kept each document in our dataset iff
np.random.pareto ()>1document_score

Language Models are Few-Shot Learners

Summary:
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - something which
current NLP systems still largely struggle to do. Here we show that scaling up
language models greatly improves task-agnostic, few-shot performance, sometimes
even reaching competitiveness with prior state-of-the-art fine-tuning
approaches. Specifically, we train GPT-3, an autoregressive language model with
175 billion parameters, 10x more than any previous non-sparse language model,
and test its performance in the few-shot setting. For all tasks, GPT-3 is
applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require
on-the-fly reasoning or domain adaptation, such as unscrambling words, using a
novel word in a sentence, or performing 3-digit arithmetic. At the same time,
we also identify some datasets where GPT-3's few-shot learning still struggles,
as well as some datasets where GPT-3 faces methodological issues related to
training on large web corpora. Finally, we find that GPT-3 can generate samples
of news articles which human evaluators have difficulty distinguishing from
articles written by humans. We discuss broader societal impacts of this finding
and of GPT-3 in general.

Tom B. Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell
Sandhini Agarwal
Ariel Herbert-Voss
Gretchen Krueger
Tom Henighan
Rewon Child
Aditya Ramesh
Daniel M. Ziegler
Jeffrey Wu
Clemens Winter
Christopher Hesse
Mark Chen
Eric Sigler
Mateusz Litwin
Scott Gray
Benjamin Chess
Jack Clark
Christopher Berner
Sam McCandlish
Alec Radford
Ilya Sutskever
Dario Amodei

40+32 pages

Learning to learn by gradient descent by gradient descent

Massively multilingual neural machine translation

Roee Aharoni, Melvin Johnson, Orhan Firat

Language (technology) is power: A critical survey of “bias” in NLP


this classiﬁer to score Common Crawl documents. We kept each document in our dataset iff
np.random.pareto ()>1document_score
We chose= 9in order to take mostly documents the classiﬁer scored highly, but still include some documents
that were out of distribution. was chosen to match the distribution of scores from our classiﬁer on WebText.
We found this re-weighting increased quality as measured by loss on a range of out-of-distribution generative
text samples.
2.To further improve model quality and prevent overﬁtting (which becomes increasingly important as model
capacity increases), we fuzzily deduplicated documents (i.e. removed documents with high overlap with
other documents) within each dataset using Spark’s MinHashLSH implementation with 10 hashes, using the
same features as were used for classiﬁcation above. We also fuzzily removed WebText from Common Crawl.
Overall this decreased dataset size by an average of 10%.
After ﬁltering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in
Appendix C.
B Details of Model Training

Language Models are Few-Shot Learners

40+32 pages


After ﬁltering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in
Appendix C.
B Details of Model Training
To train all versions of GPT-3, we use Adam with 1= 0:9,2= 0:95, and= 108, we clip the global norm of the
gradient at 1.0, and we use cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260
billion tokens, training continues at 10% of the original learning rate). There is a linear LR warmup over the ﬁrst 375
million tokens. We also gradually increase the batch size linearly from a small value (32k tokens) to the full value over
the ﬁrst 4-12 billion tokens of training, depending on the model size. Data are sampled without replacement during
training (until an epoch boundary is reached) to minimize overﬁtting. All models use weight decay of 0.1 to provide a
small amount of regularization [LH17].
During training we always train on sequences of the full nctx= 2048 token context window, packing multiple
documents into a single sequence when documents are shorter than 2048, in order to increase computational efﬁciency.

Language Models are Few-Shot Learners

40+32 pages


documents into a single sequence when documents are shorter than 2048, in order to increase computational efﬁciency.
Sequences with multiple documents are not masked in any special way but instead documents within a sequence
are delimited with a special end of text token, giving the language model the information necessary to infer that
context separated by the end of text token is unrelated. This allows for efﬁcient training without need for any special
sequence-speciﬁc masking.
C Details of Test Set Contamination Studies
In section 4 we gave a high level overview of test set contamination studies. In this section we provide details on
methodology and results.
Initial training set ﬁltering We attempted to remove text occurring in benchmarks from training data by searching
for13gram overlaps between all test/development sets used in this work and our training data, and we removed
the colliding 13gram as well as a 200 character window around it, splitting the original document into pieces. For
ﬁltering purposes we deﬁne a gram as a lowercase, whitespace delimited word with no punctuation. Pieces less than
200characters long were discarded. Documents split into more than 10 pieces were considered contaminated and

Language Models are Few-Shot Learners

40+32 pages


200characters long were discarded. Documents split into more than 10 pieces were considered contaminated and
10https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.feature.HashingTF
43
removed entirely. Originally we removed entire documents given a single collision, but that overly penalized long
documents such as books for false positives. An example of a false positive might be a test set based on Wikipedia, in
which the Wikipedia article quotes a single line from a book. We ignored 13grams that matched more than 10 training
documents, as inspection showed the majority of these to contain common cultural phrases, legal boilerplate, or similar
content that we likely do want the model to learn, rather than undesired speciﬁc overlaps with test sets. Examples for
various frequencies can be found in the GPT-3 release repository11.
Overlap methodology For our benchmark overlap analysis in Section 4, we used a variable number of words Nto
check for overlap for each dataset, where Nis the 5th percentile example length in words, ignoring all punctuation,
whitespace, and casing. Due to spurious collisions at lower values of Nwe use a minimum value of 8 on non-synthetic

Language Models are Few-Shot Learners

40+32 pages


whitespace, and casing. Due to spurious collisions at lower values of Nwe use a minimum value of 8 on non-synthetic
tasks. For performance reasons, we set a maximum value of 13 for all tasks. Values for Nand the amount of data
marked as dirty are shown in Table C.1. Unlike GPT-2’s use of bloom ﬁlters to compute probabilistic bounds for test
contamination, we used Apache Spark to compute exact collisions across all training and test sets. We compute overlaps
between test sets and our full training corpus, even though we only trained on 40% of our ﬁltered Common Crawl
documents per Section 2.2.
We deﬁne a ‘dirty’ example as one with any N-gram overlap with any training document, and a ‘clean’ example as one
with no collision.
Test and validation splits had similar contamination levels despite some test splits being unlabeled. Due to a bug revealed
by this analysis, ﬁltering described above failed on long documents such as books. Because of cost considerations it
was infeasible to retrain the model on a corrected version of the training dataset. As such, several language modeling
benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this

Language Models are Few-Shot Learners

40+32 pages


benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this
paper. Overlaps are shown in Table C.1
Overlap results To understand how much having seen some of the data helps the model perform on downstream
tasks, we ﬁlter every validation and test set by dirtiness. Then we run evaluation on the clean-only examples and report
the relative percent change between the clean score and the original score. If the clean score is more than 1% or 2%
worse than the overall score, it suggests the model may have overﬁt to the examples it has seen. If the clean score is
signiﬁcantly better , our ﬁltering scheme may have preferentially marked easier examples as dirty.
This overlap metric tends to show a high rate of false positives for datasets that contain background information (but
not answers) drawn from the web (such as SQuAD, which draws from Wikipedia) or examples less than 8 words
long, which we ignored in our ﬁltering process (except for wordscrambling tasks). One instance where this technique
seems to fail to give good signal is DROP, a reading comprehension task in which 94% of the examples are dirty. The
information required to answer the question is in a passage provided to the model, so having seen the passage during

Language Models are Few-Shot Learners

40+32 pages


information required to answer the question is in a passage provided to the model, so having seen the passage during
training but not the questions and answers does not meaningfully constitute cheating. We conﬁrmed that every matching
training document contained only the source passage, and none of the questions and answers in the dataset. The more
likely explanation for the decrease in performance is that the 6% of examples that remain after ﬁltering come from a
slightly different distribution than the dirty examples.
Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but
there is no apparent bias towards improved or degraded performance. This suggests that GPT-3 is relatively insensitive
to contamination. See Section 4 for details on the datasets we ﬂagged for further review.
11https://github.com/openai/gpt-3/blob/master/overlap_frequency.md
44
Name Split Metric N Acc/F1/BLEUTotal
CountDirty
Acc/F1/BLEUDirty
CountClean
Acc/F1/BLEUClean
CountClean
PercentageRelative
Difference
Clean vs All
Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20%

Language Models are Few-Shot Learners

40+32 pages


Language Models are Few-Shot Learners

40+32 pages


SAT Analogies dev acc 9 65.8 374 100.0 2 65.6 372 99% 0%
StoryCloze test acc 13 87.7 1871 100.0 2 87.6 1869 100% 0%
Winogrande dev acc 13 77.7 1267 - 0 77.7 1267 100% 0%
Table C.1: Overlap statistics for all datasets sorted from dirtiest to cleanest. We consider a dataset example dirty if it
has a singleN-gram collision with any document in our training corpus. “Relative Difference Clean vs All” shows the
percent change in performance between only the clean examples vs all the examples in the benchmark. “Count” shows
the number of examples. “Clean percentage” is the percent of examples that are clean vs total. For “Acc/F1/BLEU” we
use the metric speciﬁed in “Metric”. These scores come from evaluations with a different seed for the random examples
used for in-context learning, and will therefore differ slightly from the scores elsewhere in the paper.
45
D Total Compute Used to Train Language Models
This appendix contains the calculations that were used to derive the approximate compute used to train the language
models in Figure 2.2. As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10%

Language Models are Few-Shot Learners

40+32 pages


models in Figure 2.2. As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10%
of the total compute for the models we are analyzing.
Calculations can be seen in Table D.1 and are explained within the table caption.
ModelTotal train
compute
(PF-days)Total train
compute
(ﬂops)Params
(M)Training tokens
(billions)Flops
per param
per tokenMult for
bwd passFwd-pass
ﬂops per
active param
per tokenFrac of
params active
for each
token
T5-Small 2.08E+00 1.80E+20 60 1,000 3 3 1 0.5
T5-Base 7.64E+00 6.60E+20 220 1,000 3 3 1 0.5
T5-Large 2.67E+01 2.31E+21 770 1,000 3 3 1 0.5
T5-3B 1.04E+02 9.00E+21 3,000 1,000 3 3 1 0.5
T5-11B 3.82E+02 3.30E+22 11,000 1,000 3 3 1 0.5

Language Models are Few-Shot Learners

40+32 pages


GPT-3 2.7B 5.52E+01 4.77E+21 2,650 300 6 3 2 1.0
GPT-3 6.7B 1.39E+02 1.20E+22 6,660 300 6 3 2 1.0
GPT-3 13B 2.68E+02 2.31E+22 12,850 300 6 3 2 1.0
GPT-3 175B 3.64E+03 3.14E+23 174,600 300 6 3 2 1.0
Table D.1: Starting from the right hand side and moving left, we begin with the number of training tokens that each
model was trained with. Next we note that since T5 uses an encoder-decoder model, only half of the parameters are
active for each token during a forward or backwards pass. We then note that each token is involved in a single addition
and a single multiply for each active parameter in the forward pass (ignoring attention). Then we add a multiplier of
3x to account for the backwards pass (as computing both@params
@lossand@acts
@lossuse a similar amount of compute as the
forwards pass. Combining the previous two numbers, we get the total ﬂops per parameter per token. We multiply this

Language Models are Few-Shot Learners

40+32 pages


@lossuse a similar amount of compute as the
forwards pass. Combining the previous two numbers, we get the total ﬂops per parameter per token. We multiply this
value by the total training tokens and the total parameters to yield the number of total ﬂops used during training. We
report both ﬂops and petaﬂop/s-day (each of which are 8.64e+19 ﬂops).
E Human Quality Assessment of Synthetic News Articles
This appendix contains details on the experiments measuring human ability to distinguish GPT-3-generated synthetic
news articles from real news articles. We ﬁrst describe the experiments on the 200word news articles, and then
describe the preliminary investigation of 500word news articles generated by GPT-3.
Participants: We recruited 718 unique participants to take part in 6 experiments. 97 participants were excluded for
failing an internet check question, leaving a total of 621 participants: 343 male, 271 female, and 7 other. Mean
participant age was 38years old. All participants were recruited through Positly, which maintains a whitelist of
high-performing workers from Mechanical Turk. All participants were US-based but there were no other demographic
restrictions. Participants were paid $12 for their participation, based on a task time estimate of 60 minutes determined

Language Models are Few-Shot Learners

40+32 pages


restrictions. Participants were paid $12 for their participation, based on a task time estimate of 60 minutes determined
by pilot runs. In order to ensure that the sample of participants for each experiment quiz was unique, participants were
not allowed to take part in an experiment more than once.
Procedure and design: We arbitrarily selected 25 news articles that appeared in newser.com in early 2020. We used
the article titles and subtitles to produce outputs from the 125M, 350M, 760M, 1.3B, 2.7B, 6.7B, 13.0B, and 200B
(GPT-3) parameter language models. Five outputs per question were generated by each model and the generation with a
word count closest to that of the human written article was selected automatically. This was to minimize the effect
that completion length might have on participants’ judgments. The same output procedure for each model with the
exception of the removal of the intentionally bad control model, as described in the main text.
46
ModelParticipants
RecruitedParticipants
ExcludedGenders
(m:f:other)Mean
AgeAverage
Word Count
(human:model)
Control 76 7 32:37:0 39 216:216
GPT-3 Small 80 7 41:31:1 40 216:188

Language Models are Few-Shot Learners

40+32 pages


Word Count
(human:model)
Control 76 7 32:37:0 39 216:216
GPT-3 Small 80 7 41:31:1 40 216:188
GPT-3 Medium 80 7 46:28:2 39 216:202
GPT-3 Large 81 24 46:28:2 37 216:200
GPT-3 XL 79 14 32:32:1 38 216:199
GPT-3 2.7B 80 11 36:33:0 40 216:202
GPT-3 6.7B 76 5 46:28:2 37 216:195
GPT-3 13.0B 81 13 46:28:2 37 216:209
GPT-3 175B 80 9 42:29:0 37 216:216
Table E.1: Participant details and article lengths for each experiment to evaluate human detection of 200word model
generated news articles. Participants were excluded due to internet check fails.
Figure E.1: Participants spend more time trying to identify whether each news article is machine generated as model
size increases. Duration on the control model is indicated with the dashed line. Line of best ﬁt is a linear model on a log
scale with 95% conﬁdence intervals.
In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz

Language Models are Few-Shot Learners

40+32 pages


scale with 95% conﬁdence intervals.
In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz
B. Each quiz consisted of 25 articles: half (12-13) were human written and half (12-13) were model generated: the
articles with human written completions in quiz A had model generated completions in quiz B and vice versa. The
order of quiz question was shufﬂed for each participant. Participants could leave comments and were asked to indicate
if they had seen the articles before. Participants were instructed not to look up the articles or their content during the
quiz and at the end of the quiz were asked if they had looked anything up during the quiz.
Statistical Tests: To compare means on the different runs, we performed a two-sample t-test for independent groups for
each model against the control. This was implemented in Python using the scipy.stats.ttest_ind function. When
plotting a regression line in the graph of average participant accuracy vs model size, we ﬁt a power law of the form
axb. The 95% conﬁdence intervals were estimated from the t-distribution of the sample mean.
Duration statistics : In the main text, we discussed the ﬁnding that the ability of human participants to distinguish

Language Models are Few-Shot Learners

40+32 pages


Duration statistics : In the main text, we discussed the ﬁnding that the ability of human participants to distinguish
model and human generated news articles decreases as our models become larger. We have also found that the
average time spent for a given set of questions increases as the model size increases, as shown in Figure E.1. Lower
47
ModelParticipants
RecruitedParticipants
ExcludedGenders
(m:f:other)Mean
AgeAverage
Word Count
(human:model)
Control 79 17 32:37:0 39 569:464
GPT-3 175B 81 19 32:30:0 40 569:498
Table E.2: Participant details and article lengths for the experiments investigating human detection of 500word
model generated news articles. Participants were excluded due to internet check fails.
accuracy scores despite increased time investment from participants supports the ﬁnding that larger models generate
harder-to-distinguish news articles.
Preliminary investigation of 500word articles: We recruited 160 unique US-based participants to take part in 2
experiments through Positly (details are given in Table E.2). We randomly selected 12 Reuters world news articles from
late 2019 and created a context for GPT-3 175B that consisted of a single Reuters article not in this set of 12. We then

Language Models are Few-Shot Learners

40+32 pages


late 2019 and created a context for GPT-3 175B that consisted of a single Reuters article not in this set of 12. We then
used the article titles and Reuters locations to generate completions from GPT-3 175B and the 160M control model
from the previous experiments. These were used to create two 12-question quizzes per model, each consisting of half
human written and half model generated articles. Comprehension questions were added and articles were shown to
participants in 3 stages at 30 second intervals to encourage closer reading. Participants were paid $12 for this task.
Model generation selection methods, exclusion criteria, and statistical tests mirror those of the previous experiments.
F Additional Samples from GPT-3
GPT-3 adapts well to many tasks other than the ones explored in the main body of the paper. As an example, in Figure
F.1, we show four uncurated samples from a prompt suggesting that the model write a poem, with a given title, in the
style of Wallace Stevens. We ﬁrst experimented with a few prompts, then generated four samples with no additional
editing or selection (sampling at temperature 1using nucleus sampling [ HBFC19 ] withP= 0:9). Completions were
truncated when the model began to write a new title and author heading, or broke into prose commentary.
48
Context! The City

Language Models are Few-Shot Learners

40+32 pages


truncated when the model began to write a new title and author heading, or broke into prose commentary.
48
Context! The City
BY C. P. CAVAFY
TRANSLATED BY EDMUND KEELEY
[Poem text omitted]
SOME TREES
John Ashbery
[Poem text omitted]
Shadows on the Way
Wallace Stevens
-------- Generated Poem 1 -------- -------- Generated Poem 3 -------I must have shadows on the way The sun was all we had. Now, in the shade
If I am to walk I must have All is changed. The mind must dwell on those
Each step taken slowly and alone White fields, that to its eyes were always old;
To have it ready made Those ancient gleams, convoluting
The way they lay among the huge roots,
And I must think in lines of grey The great dark tomes of reverie,
To have dim thoughts to be my guide The plumed labyrinth of the sea.
Must look on blue and green
And never let my eye forget When our own breath gonne faint and fade
That color is my friend Up the long stairway of the dead
And purple must surround me too We shall remember well
The blindness of the light, the whiteness
The yellow of the sun is no more Of that white land. The footsteps, and

Language Models are Few-Shot Learners

40+32 pages


And purple must surround me too We shall remember well
The blindness of the light, the whiteness
The yellow of the sun is no more Of that white land. The footsteps, and
Intrusive than the bluish snow The grasses in that light, and in the shade
That falls on all of us. I must have The menacing life.
Grey thoughts and blue thoughts walk with me
If I am to go away at all. We played, too, once, in the warmest rooms.
To one content, there is one content
-------- Generated Poem 2 -------- For all mankind. The forest glades
Are all the more for being in fable,
There is a way in the world. And wood and town in legend shadowedThat way a man may live in a small town Once more to return. The one content
His lifetime and see no man come, Has in it every happiness, more brave
Never the stir of men about a market, For being purposeless; and so
Never the bobbing of covered carts, Those island gardens, deep in the mind,
Never the creak passed on the stair Make of remoteness familiarity
Closing some other life; an old man And of the unspoken name a friend.
Can prosper in that town, can prosper

Language Models are Few-Shot Learners

40+32 pages


Closing some other life; an old man And of the unspoken name a friend.
Can prosper in that town, can prosper
When his store is seen to close, his wife -------- Generated Poem 4 -------Fades in his house, and the world shouts in vain
That no one comes to hear him Nobody will come to this place. It is a road that leads nowhere.
Where his small bell tolls. The solitude is deep. The mountains are high.
In that small town But they are desolate, and they turn the traveler's face
Everything is known. Towards the North. All the sounds of the world are far away.
The ways of the young men are known, When the wind rises above the trees,
And the ways of the young women The boughs bow to the ground.
As they walk to the Spring; Even the birds that inhabit the tangle of weeds
The rumors are all known, That is the roadside cover, are silent. One listens,
The quarrels, the hurt, the envy But hears no roar of the forest. One is alone.
Of the men and the women. One will be taken.
Till a day comes One will be taken.
That way out of the world There is no utterance, there is no conversation,
That leads to a wood. But one is uneasy all the same....
Young, the young man There is a thin blue mist,

Language Models are Few-Shot Learners

40+32 pages


That leads to a wood. But one is uneasy all the same....
Young, the young man There is a thin blue mist,
Walks, a lover now, A darkness rising like smoke,
Knows the streets as they are and is of them. And within that darkness
A possession of the heart.
Comes to the wood. One will be taken.... It was here, and it will be here againHere, under this sky empty and full of light.
There, in the wood, among trees,
He sees shadows on the way, hears voices,
hears the wind and the rustling of leaves;
Through an open glade
He sees a shape and the shape hears:
It waits as he waits,
Waits as the shadows wait,
As the voices wait;
Shadows on the way, voices in the wind.
Figure F.1: Four uncurated completions from a context suggesting the model compose a poem in the style of Wallace
Stevens with the title ‘Shadows on the Way’.
49
G Details of Task Phrasing and Speciﬁcations
The following ﬁgures illustrate the formatting and phrasing of all the tasks included in the paper. All data comes from
the ground truth datasets in this section, and no samples from GPT-3 are included here.
Context!Article:

Language Models are Few-Shot Learners

40+32 pages


the ground truth datasets in this section, and no samples from GPT-3 are included here.
Context!Article:
Informal conversation is an important part of any business
relationship.Before you start a discussion,however,make sure you understand
which topics are suitable and which are considered taboo in a particular
culture. Latin Americans enjoy sharing information about their local
history, art and customs.You may expect questions about your family,and
be sure to show pictures of your children.You may feel free to ask similar
questions of your Latin American friends.The French think of conversation
as an art form,and they enjoy the value of lively discussions as well as
disagreements. For them,arguments can be interesting and they can cover
pretty much or any topic ---- as long as they occur in are respectful and
intelligent manner.
In the United States,business people like to discuss a wide range of
topics,including opinions about work,family,hobbies,and politics. In
Japan,China,and Korea,however,people are much more private.They do not
share much about their thoughts,feelings,or emotions because they feel
that doing so might take away from the harmonious business relationship
they're trying to build.Middle Easterners are also private about their
personal lives and family matters.It is considered rude,for example,to ask

Language Models are Few-Shot Learners

40+32 pages


they're trying to build.Middle Easterners are also private about their
personal lives and family matters.It is considered rude,for example,to ask
a businessman from Saudi Arabia about his wife or children.
As a general rule,it's best not to talk about politics or religion with
your business friends.This can get you into trouble,even in the United
States,where people hold different religious views.In addition,discussing
one's salary is usually considered unsuitable.Sports is typically a
friendly subject in most parts of the world,although be careful not to
criticize national sport.Instead,be friendly and praise your host's team.
Q: What shouldn't you do when talking about sports with colleagues from
another country?
A: Criticizing the sports of your colleagues' country.
Q: Which is typically a friendly topic in most places according to the
author?
A: Sports.
Q: Why are people from Asia more private in their conversation with others?
A: They don't want to have their good relationship with others harmed by
informal conversation.
Q: The author considers politics and religion .
A:
Correct Answer !taboo
Incorrect Answer !cheerful topics
Incorrect Answer !rude topics
Incorrect Answer !topics that can never be talked about

Language Models are Few-Shot Learners

40+32 pages


A:
Correct Answer !taboo
Incorrect Answer !cheerful topics
Incorrect Answer !rude topics
Incorrect Answer !topics that can never be talked about
Figure G.1: Formatted dataset example for RACE-h. When predicting, we normalize by the unconditional probability
of each answer as described in 2.
50
Context!anli 2: anli 2: The Gold Coast Hotel & Casino is a hotel and casino
located in Paradise, Nevada. This locals' casino is owned and operated
by Boyd Gaming. The Gold Coast is located one mile (  1:6km) west of the
Las Vegas Strip on West Flamingo Road. It is located across the street
from the Palms Casino Resort and the Rio All Suite Hotel and Casino.
Question: The Gold Coast is a budget-friendly casino. True, False, or
Neither?
Correct Answer !Neither
Incorrect Answer !True
Incorrect Answer !False
Figure G.2: Formatted dataset example for ANLI R2
Context!Article:
Mrs. Smith is an unusual teacher. Once she told each student to bring
along a few potatoes in plastic bag. On each potato the students had to
write a name of a person that they hated And the next day, every child
brought some potatoes. Some had two potatoes;some three;some up to five.

Language Models are Few-Shot Learners

40+32 pages


write a name of a person that they hated And the next day, every child
brought some potatoes. Some had two potatoes;some three;some up to five.
Mrs. Smith then told the children to carry the bags everywhere they went,
even to the toilet, for two weeks. As day after day passed, the children
started to complain about the awful smell of the rotten potatoes.
Those children who brought five potatoes began to feel the weight trouble
of the bags. After two weeks, the children were happy to hear that the
game was finally ended. Mrs. Smith asked,"How did you feel while carrying
the potatoes for two weeks?" The children started complaining about the
trouble loudly.
Then Mrs. Smith told them why she asked them to play the game. She
said,"This is exactly the situation when you carry your hatred for somebody
inside your heart. The terrible smell of the hatred will pollute your
heart and you will carry something unnecessary with you all the time. If
you cannot stand the smell of the rotten potatoes for just two weeks, can
you imagine how heavy it would be to have the hatred in your heart for your
lifetime? So throw away any hatred from your heart, and you'll be really
happy."
Q: Which of the following is True according to the passage?
A: If a kid hated four people,he or she had to carry four potatoes.

Language Models are Few-Shot Learners

40+32 pages


happy."
Q: Which of the following is True according to the passage?
A: If a kid hated four people,he or she had to carry four potatoes.
Q: We can learn from the passage that we should .
A: throw away the hatred inside
Q: The children complained about besides the weight trouble.
A: the smell
Q: Mrs.Smith asked her students to write on the potatoes.
A:
Correct Answer !names
Incorrect Answer !numbers
Incorrect Answer !time
Incorrect Answer !places
Figure G.3: Formatted dataset example for RACE-m. When predicting, we normalize by the unconditional probability
of each answer as described in 2.
51
Context!How to apply sealant to wood.
Correct Answer !Using a brush, brush on sealant onto wood until it is fully saturated with
the sealant.
Incorrect Answer !Using a brush, drip on sealant onto wood until it is fully saturated with
the sealant.
Figure G.4: Formatted dataset example for PIQA
Context!My body cast a shadow over the grass because
Correct Answer !the sun was rising.
Incorrect Answer !the grass was cut.
Figure G.5: Formatted dataset example for COPA
Context!(CNN) Yuval Rabin, whose father, Yitzhak Rabin, was assassinated while

Language Models are Few-Shot Learners

40+32 pages


Figure G.5: Formatted dataset example for COPA
Context!(CNN) Yuval Rabin, whose father, Yitzhak Rabin, was assassinated while
serving as Prime Minister of Israel, criticized Donald Trump for appealing
to "Second Amendment people" in a speech and warned that the words that
politicians use can incite violence and undermine democracy. "Trump's
words are an incitement to the type of political violence that touched
me personally," Rabin wrote in USAToday. He said that Trump's appeal to
"Second Amendment people" to stop Hillary Clinton -- comments that were
criticized as a call for violence against Clinton, something Trump denied
-- "were a new level of ugliness in an ugly campaign season."
- The son of a former Israeli Prime Minister who was assassinated wrote an
op ed about the consequence of violent political rhetoric.
- Warns of "parallels" between Israel of the 1990s and the U.S. today.
Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Donald Trump's
aggressive rhetoric.
Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Trump's aggressive
rhetoric.

Language Models are Few-Shot Learners

40+32 pages


Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Trump's aggressive
rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Hillary Clinton's
aggressive rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned U.S.'s aggressive
rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Yitzhak Rabin's
aggressive rhetoric.
Figure G.6: Formatted dataset example for ReCoRD. We consider the context above to be a single ”problem” because
this is how the task is presented in the ReCoRD dataset and scored in the ReCoRD evaluation script.
Context!anli 1: anli 1: Fulton James MacGregor MSP is a Scottish politician
who is a Scottish National Party (SNP) Member of Scottish Parliament
for the constituency of Coatbridge and Chryston. MacGregor is currently
Parliamentary Liaison Officer to Shona Robison, Cabinet Secretary for

Language Models are Few-Shot Learners

40+32 pages


for the constituency of Coatbridge and Chryston. MacGregor is currently
Parliamentary Liaison Officer to Shona Robison, Cabinet Secretary for
Health & Sport. He also serves on the Justice and Education & Skills
committees in the Scottish Parliament.
Question: Fulton James MacGregor is a Scottish politican who is a Liaison
officer to Shona Robison who he swears is his best friend. True, False, or
Neither?
Correct Answer !Neither
Incorrect Answer !True
Incorrect Answer !False
Figure G.7: Formatted dataset example for ANLI R1
52
Context!Organisms require energy in order to do what?
Correct Answer !mature and develop.
Incorrect Answer !rest soundly.
Incorrect Answer !absorb light.
Incorrect Answer !take in nutrients.
Figure G.8: Formatted dataset example for OpenBookQA. When predicting, we normalize by the unconditional
probability of each answer as described in 2.
Context!Making a cake: Several cake pops are shown on a display. A woman and girl
are shown making the cake pops in a kitchen. They
Correct Answer !bake them, then frost and decorate.
Incorrect Answer !taste them as they place them on plates.
Incorrect Answer !put the frosting on the cake as they pan it.

Language Models are Few-Shot Learners

40+32 pages


Incorrect Answer !taste them as they place them on plates.
Incorrect Answer !put the frosting on the cake as they pan it.
Incorrect Answer !come out and begin decorating the cake as well.
Figure G.9: Formatted dataset example for HellaSwag
Context!anli 3: anli 3: We shut the loophole which has American workers actually
subsidizing the loss of their own job. They just passed an expansion of
that loophole in the last few days: $43 billion of giveaways, including
favors to the oil and gas industry and the people importing ceiling fans
from China.
Question: The loophole is now gone True, False, or Neither?
Correct Answer !False
Incorrect Answer !True
Incorrect Answer !Neither
Figure G.10: Formatted dataset example for ANLI R3
Context!Question: George wants to warm his hands quickly by rubbing them. Which
skin surface will produce the most heat?
Answer:
Correct Answer !dry palms
Incorrect Answer !wet palms
Incorrect Answer !palms covered with oil
Incorrect Answer !palms covered with lotion
Figure G.11: Formatted dataset example for ARC (Challenge). When predicting, we normalize by the unconditional
probability of each answer as described in 2.
Context!lull is to trust as

Language Models are Few-Shot Learners

40+32 pages


probability of each answer as described in 2.
Context!lull is to trust as
Correct Answer !cajole is to compliance
Incorrect Answer !balk is to fortitude
Incorrect Answer !betray is to loyalty
Incorrect Answer !hinder is to destination
Incorrect Answer !soothe is to passion
Figure G.12: Formatted dataset example for SAT Analogies
Correct Context !Grace was happy to trade me her sweater for my jacket. She thinks the
sweater
Incorrect Context !Grace was happy to trade me her sweater for my jacket. She thinks the
jacket
Target Completion !looks dowdy on her.
Figure G.13: Formatted dataset example for Winograd. The ‘partial’ evaluation method we use compares the probability
of the completion given a correct and incorrect context.
53
Correct Context !Johnny likes fruits more than vegetables in his new keto diet because the
fruits
Incorrect Context !Johnny likes fruits more than vegetables in his new keto diet because the
vegetables
Target Completion !are saccharine.
Figure G.14: Formatted dataset example for Winogrande. The ‘partial’ evaluation method we use compares the
probability of the completion given a correct and incorrect context.
Context!READING COMPREHENSION ANSWER KEY

Language Models are Few-Shot Learners

40+32 pages


probability of the completion given a correct and incorrect context.
Context!READING COMPREHENSION ANSWER KEY
While this process moved along, diplomacy continued its rounds. Direct
pressure on the Taliban had proved unsuccessful. As one NSC staff note
put it, "Under the Taliban, Afghanistan is not so much a state sponsor
of terrorism as it is a state sponsored by terrorists." In early 2000,
the United States began a high-level effort to persuade Pakistan to use
its influence over the Taliban. In January 2000, Assistant Secretary
of State Karl Inderfurth and the State Department's counterterrorism
coordinator, Michael Sheehan, met with General Musharraf in Islamabad,
dangling before him the possibility of a presidential visit in March as a
reward for Pakistani cooperation. Such a visit was coveted by Musharraf,
partly as a sign of his government's legitimacy. He told the two envoys
that he would meet with Mullah Omar and press him on Bin Laden. They
left, however, reporting to Washington that Pakistan was unlikely in fact
to do anything," given what it sees as the benefits of Taliban control
of Afghanistan." President Clinton was scheduled to travel to India.
The State Department felt that he should not visit India without also
visiting Pakistan. The Secret Service and the CIA, however, warned in
the strongest terms that visiting Pakistan would risk the President's

Language Models are Few-Shot Learners

40+32 pages


The State Department felt that he should not visit India without also
visiting Pakistan. The Secret Service and the CIA, however, warned in
the strongest terms that visiting Pakistan would risk the President's
life. Counterterrorism officials also argued that Pakistan had not done
enough to merit a presidential visit. But President Clinton insisted
on including Pakistan in the itinerary for his trip to South Asia. His
one-day stopover on March 25, 2000, was the first time a U.S. president
had been there since 1969. At his meeting with Musharraf and others,
President Clinton concentrated on tensions between Pakistan and India
and the dangers of nuclear proliferation, but also discussed Bin Laden.
President Clinton told us that when he pulled Musharraf aside for a brief,
one-on-one meeting, he pleaded with the general for help regarding Bin
Laden." I offered him the moon when I went to see him, in terms of better
relations with the United States, if he'd help us get Bin Laden and deal
with another issue or two." The U.S. effort continued.
Who did The State Department feel should visit both India and Pakistan?
Correct Answer !- [False] Bin Laden
Incorrect Answer !- [True] Bin Laden
Figure G.15: Formatted dataset example for MultiRC. There are three levels within MultiRC: (1) the passage, (2) the

Language Models are Few-Shot Learners

40+32 pages


Figure G.15: Formatted dataset example for MultiRC. There are three levels within MultiRC: (1) the passage, (2) the
questions, and (3) the answers. During evaluation, accuracy is determined at the per-question level, with a question
being considered correct if and only if all the answers within the question are labeled correctly. For this reason, we use
Kto refer to the number of questions shown within the context.
Context!Question: Which factor will most likely cause a person to develop a fever?
Answer:
Correct Answer !a bacterial population in the bloodstream
Incorrect Answer !a leg muscle relaxing after exercise
Incorrect Answer !several viral particles on the skin
Incorrect Answer !carbohydrates being digested in the stomach
Figure G.16: Formatted dataset example for ARC (Easy). When predicting, we normalize by the unconditional
probability of each answer as described in 2.
54
Context!Bob went to the gas station to fill up his car. His tank was completely
empty and so was his wallet. The cashier offered to pay for his gas if he
came back later to pay. Bob felt grateful as he drove home.
Correct Answer !Bob believed that there were good people in the world.
Incorrect Answer !Bob contemplated how unfriendly the world was.

Language Models are Few-Shot Learners

40+32 pages


Correct Answer !Bob believed that there were good people in the world.
Incorrect Answer !Bob contemplated how unfriendly the world was.
Figure G.17: Formatted dataset example for StoryCloze
Context!Helsinki is the capital and largest city of Finland. It is in the region
of Uusimaa, in southern Finland, on the shore of the Gulf of Finland.
Helsinki has a population of , an urban population of , and a metropolitan
population of over 1.4 million, making it the most populous municipality
and urban area in Finland. Helsinki is some north of Tallinn, Estonia,
east of Stockholm, Sweden, and west of Saint Petersburg, Russia. Helsinki
has close historical connections with these three cities.
The Helsinki metropolitan area includes the urban core of Helsinki, Espoo,
Vantaa, Kauniainen, and surrounding commuter towns. It is the world's
northernmost metro area of over one million people, and the city is the
northernmost capital of an EU member state. The Helsinki metropolitan
area is the third largest metropolitan area in the Nordic countries
after Stockholm and Copenhagen, and the City of Helsinki is the third
largest after Stockholm and Oslo. Helsinki is Finland's major political,
educational, financial, cultural, and research center as well as one of
northern Europe's major cities. Approximately 75% of foreign companies

Language Models are Few-Shot Learners

40+32 pages


educational, financial, cultural, and research center as well as one of
northern Europe's major cities. Approximately 75% of foreign companies
that operate in Finland have settled in the Helsinki region. The nearby
municipality of Vantaa is the location of Helsinki Airport, with frequent
service to various destinations in Europe and Asia.
Q: what is the most populous municipality in Finland?
A: Helsinki
Q: how many people live there?
A: 1.4 million in the metropolitan area
Q: what percent of the foreign companies that operate in Finland are in
Helsinki?
A: 75%
Q: what towns are a part of the metropolitan area?
A:
Target Completion !Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns
Figure G.18: Formatted dataset example for CoQA
Context!Please unscramble the letters into a word, and write that word:
asinoc =
Target Completion !casino
Figure G.19: Formatted dataset example for Cycled Letters
55
Context!Passage: Saint Jean de Br ebeuf was a French Jesuit missionary who
travelled to New France in 1625. There he worked primarily with the Huron
for the rest of his life, except for a few years in France from 1629 to

Language Models are Few-Shot Learners

40+32 pages


travelled to New France in 1625. There he worked primarily with the Huron
for the rest of his life, except for a few years in France from 1629 to
1633. He learned their language and culture, writing extensively about
each to aid other missionaries. In 1649, Br ebeuf and another missionary
were captured when an Iroquois raid took over a Huron village . Together
with Huron captives, the missionaries were ritually tortured and killed
on March 16, 1649. Br ebeuf was beatified in 1925 and among eight Jesuit
missionaries canonized as saints in the Roman Catholic Church in 1930.
Question: How many years did Saint Jean de Br ebeuf stay in New France
before he went back to France for a few years?
Answer:
Target Completion !4
Figure G.20: Formatted dataset example for DROP
Context!Fill in blank:
She held the torch in front of her.
She caught her breath.
"Chris? There's a step."
"What?"
"A step. Cut in the rock. About fifty feet ahead." She moved faster.
They both moved faster. "In fact," she said, raising the torch higher,
"there's more than a . ->
Target Completion !step
Figure G.21: Formatted dataset example for LAMBADA

Language Models are Few-Shot Learners

40+32 pages


"there's more than a . ->
Target Completion !step
Figure G.21: Formatted dataset example for LAMBADA
Context!Please unscramble the letters into a word, and write that word:
skicts =
Target Completion !sticks
Figure G.22: Formatted dataset example for Anagrams 1 (A1)
Context!Please unscramble the letters into a word, and write that word:
volwskagen =
Target Completion !volkswagen
Figure G.23: Formatted dataset example for Anagrams 2
Context!Q: Who played tess on touched by an angel?
A:
Target Completion !Delloreese Patricia Early (July 6, 1931 { November 19, 2017), known
professionally as Della Reese
Figure G.24: Formatted dataset example for Natural Questions
56
Context!TITLE: William Perry (American football) - Professional career
PARAGRAPH: In 1985, he was selected in the first round of the 1985 NFL
Draft by the Chicago Bears; he had been hand-picked by coach Mike Ditka.
However, defensive coordinator Buddy Ryan, who had a highly acrimonious
relationship with Ditka, called Perry a "wasted draft-pick". Perry
soon became a pawn in the political power struggle between Ditka and

Language Models are Few-Shot Learners

40+32 pages


relationship with Ditka, called Perry a "wasted draft-pick". Perry
soon became a pawn in the political power struggle between Ditka and
Ryan. Perry's "Refrigerator" nickname followed him into the NFL and he
quickly became a favorite of the Chicago Bears fans. Teammates called
him "Biscuit," as in "one biscuit shy of 350 pounds." While Ryan refused
to play Perry, Ditka decided to use Perry as a fullback when the team was
near the opponents' goal line or in fourth and short situations, either
as a ball carrier or a lead blocker for star running back Walter Payton.
Ditka stated the inspiration for using Perry as a fullback came to him
during five-yard sprint exercises. During his rookie season, Perry
rushed for two touchdowns and caught a pass for one. Perry even had
the opportunity to run the ball during Super Bowl XX, as a nod to his
popularity and contributions to the team's success. The first time he
got the ball, he was tackled for a one-yard loss while attempting to throw
his first NFL pass on a halfback option play. The second time he got the
ball, he scored a touchdown (running over Patriots linebacker Larry McGrew
in the process). About halfway through his rookie season, Ryan finally
began to play Perry, who soon proved that he was a capable defensive

Language Models are Few-Shot Learners

40+32 pages


in the process). About halfway through his rookie season, Ryan finally
began to play Perry, who soon proved that he was a capable defensive
lineman. His Super Bowl ring size is the largest of any professional
football player in the history of the event. His ring size is 25, while
the ring size for the average adult male is between 10 and 12. Perry went
on to play for ten years in the NFL, retiring after the 1994 season. In
his ten years as a pro, he regularly struggled with his weight, which
hampered his performance at times. He played in 138 games, recording
29.5 sacks and five fumble recoveries, which he returned for a total of
71 yards. In his offensive career he ran five yards for two touchdowns,
and had one reception for another touchdown. Perry later attempted a
comeback, playing an unremarkable 1996 season with the London Monarchs of
the World League of American Football (later NFL Europa).
Q: what team did he play for?
A:
Target Completion !the Chicago Bears
Figure G.25: Formatted dataset example for QuAC
Context!Please unscramble the letters into a word, and write that word:
r e!c.i p r o.c a/l =
Target Completion !reciprocal
Figure G.26: Formatted dataset example for Symbol Insertion

Language Models are Few-Shot Learners

40+32 pages


r e!c.i p r o.c a/l =
Target Completion !reciprocal
Figure G.26: Formatted dataset example for Symbol Insertion
Context!Please unscramble the letters into a word, and write that word:
taefed =
Target Completion !defeat
Figure G.27: Formatted dataset example for Reversed Words
57
Context!Title: The Blitz
Background: From the German point of view, March 1941 saw an improvement.
The Luftwaffe flew 4,000 sorties that month, including 12 major and
three heavy attacks. The electronic war intensified but the Luftwaffe
flew major inland missions only on moonlit nights. Ports were easier to
find and made better targets. To confuse the British, radio silence was
observed until the bombs fell. X- and Y-Ger at beams were placed over
false targets and switched only at the last minute. Rapid frequency
changes were introduced for X-Ger at, whose wider band of frequencies and
greater tactical flexibility ensured it remained effective at a time when
British selective jamming was degrading the effectiveness of Y-Ger at.
Q: How many sorties were flown in March 1941?
A: 4,000
Q: When did the Luftwaffe fly inland missions?
A:
Target Completion !only on moonlit nights

Language Models are Few-Shot Learners

40+32 pages


A: 4,000
Q: When did the Luftwaffe fly inland missions?
A:
Target Completion !only on moonlit nights
Figure G.28: Formatted dataset example for SQuADv2
Context!Normal force -- In a simple case such as an object resting upon a table,
the normal force on the object is equal but in opposite direction to the
gravitational force applied on the object (or the weight of the object),
that is, N = m g ( ndisplaystyle N=mg), where m is mass, and g is the
gravitational field strength (about 9.81 m/s on Earth). The normal force
here represents the force applied by the table against the object that
prevents it from sinking through the table and requires that the table is
sturdy enough to deliver this normal force without breaking. However, it
is easy to assume that the normal force and weight are action-reaction
force pairs (a common mistake). In this case, the normal force and
weight need to be equal in magnitude to explain why there is no upward
acceleration of the object. For example, a ball that bounces upwards
accelerates upwards because the normal force acting on the ball is larger
in magnitude than the weight of the ball.
question: is the normal force equal to the force of gravity?
answer:
Target Completion !yes

Language Models are Few-Shot Learners

40+32 pages


in magnitude than the weight of the ball.
question: is the normal force equal to the force of gravity?
answer:
Target Completion !yes
Figure G.29: Formatted dataset example for BoolQ
Context!The trend toward lower rents may seem surprising given that some
communities in New York are bemoaning the loss of favorite local
businesses to high rents. But, despite the recent softening, for many
of these retailers there's still been too big a jump from the rental rates
of the late 1970s, when their leases were signed. Certainly, the recent
drop in prices doesn't mean Manhattan comes cheap.
question: Manhattan comes cheap. true, false, or neither?
answer:
Target Completion !false
Figure G.30: Formatted dataset example for CB
58
Context!The bet, which won him dinner for four, was regarding the existence and
mass of the top quark, an elementary particle discovered in 1995.
question: The Top Quark is the last of six flavors of quarks predicted by
the standard model theory of particle physics. True or False?
answer:
Target Completion !False
Figure G.31: Formatted dataset example for RTE
Context!An outfitter provided everything needed for the safari.
Before his first walking holiday, he went to a specialist outfitter to buy
some boots.

Language Models are Few-Shot Learners

40+32 pages


Context!An outfitter provided everything needed for the safari.
Before his first walking holiday, he went to a specialist outfitter to buy
some boots.
question: Is the word `outfitter' used in the same way in the two
sentences above?
answer:
Target Completion !no
Figure G.32: Formatted dataset example for WiC
Context!Final Exam with Answer Key
Instructions: Please carefully read the following passages. For each
passage, you must identify which noun the pronoun marked in *bold* refers
to.
=====
Passage: Mr. Moncrieff visited Chester's luxurious New York apartment,
thinking that it belonged to his son Edward. The result was that Mr.
Moncrieff has decided to cancel Edward's allowance on the ground that
he no longer requires *his* financial support.
Question: In the passage above, what does the pronoun "*his*" refer to?
Answer:
Target Completion !mr. moncrieff
Figure G.33: Formatted dataset example for WSC
Context!Q: `Nude Descending A Staircase' is perhaps the most famous painting by
which 20th century artist?
A:
Target Completion !MARCEL DUCHAMP
Target Completion !r mutt
Target Completion !duchamp

Language Models are Few-Shot Learners

40+32 pages


which 20th century artist?
A:
Target Completion !MARCEL DUCHAMP
Target Completion !r mutt
Target Completion !duchamp
Target Completion !marcel duchamp
Target Completion !R.Mutt
Target Completion !Marcel duChamp
Target Completion !Henri-Robert-Marcel Duchamp
Target Completion !Marcel du Champ
Target Completion !henri robert marcel duchamp
Target Completion !Duchampian
Target Completion !Duchamp
Target Completion !duchampian
Target Completion !marcel du champ
Target Completion !Marcel Duchamp
Target Completion !MARCEL DUCHAMP
Figure G.34: Formatted dataset example for TriviaQA. TriviaQA allows for multiple valid completions.
59
Context!Q: What school did burne hogarth establish?
A:
Target Completion !School of Visual Arts
Figure G.35: Formatted dataset example for WebQA
Context!Keinesfalls d urfen diese f ur den kommerziellen Gebrauch verwendet werden.
=
Target Completion !In no case may they be used for commercial purposes.

Language Models are Few-Shot Learners

40+32 pages


=
Target Completion !In no case may they be used for commercial purposes.
Figure G.36: Formatted dataset example for De !En. This is the format for one- and few-shot learning, for this and
other langauge tasks, the format for zero-shot learning is “Q: What is the flanguagegtranslation offsentencegA:
ftranslationg.”
Context!In no case may they be used for commercial purposes. =
Target Completion !Keinesfalls d urfen diese f ur den kommerziellen Gebrauch verwendet werden.
Figure G.37: Formatted dataset example for En !De
Context!Analysis of instar distributions of larval I. verticalis collected from
a series of ponds also indicated that males were in more advanced instars
than females. =
Target Completion !L'analyse de la distribution de fr equence des stades larvaires d'I.
verticalis dans une s erie d' etangs a  egalement d emontr e que les larves
m^ ales  etaient  a des stades plus avanc es que les larves femelles.
Figure G.38: Formatted dataset example for En !Fr

Language Models are Few-Shot Learners

40+32 pages


Figure G.38: Formatted dataset example for En !Fr
Context!L'analyse de la distribution de fr equence des stades larvaires d'I.
verticalis dans une s erie d' etangs a  egalement d emontr e que les larves
m^ ales  etaient  a des stades plus avanc es que les larves femelles. =
Target Completion !Analysis of instar distributions of larval I. verticalis collected from
a series of ponds also indicated that males were in more advanced instars
than females.
Figure G.39: Formatted dataset example for Fr !En
Context!The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession
to the European Union, despite Turkey's continuing refusal to recognise
Cyprus and despite the fact that the democratic reforms are at a
standstill. =
Target Completion !Adev arul este c a v a dorit i, cu orice pret   si ^ mpotriva dorint ei
europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea

Language Models are Few-Shot Learners

40+32 pages


europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea
European a, ^ n ciuda refuzului continuu al Turciei de a recunoa ste Ciprul
 si ^ n ciuda faptului c a reformele democratice au ajuns ^ ntr-un punct mort.
Figure G.40: Formatted dataset example for En !Ro
60
Context!Adev arul este c a v a dorit i, cu orice pret   si ^ mpotriva dorint ei
europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea
European a, ^ n ciuda refuzului continuu al Turciei de a recunoa ste Ciprul
 si ^ n ciuda faptului c a reformele democratice au ajuns ^ ntr-un punct mort.
=
Target Completion !The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession

Language Models are Few-Shot Learners

40+32 pages


=
Target Completion !The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession
to the European Union, despite Turkey's continuing refusal to recognise
Cyprus and despite the fact that the democratic reforms are at a
standstill.
Figure G.41: Formatted dataset example for Ro !En
Context!Q: What is (2 * 4) * 6?
A:
Target Completion !48
Figure G.42: Formatted dataset example for Arithmetic 1DC
Context!Q: What is 17 minus 14?
A:
Target Completion !3
Figure G.43: Formatted dataset example for Arithmetic 2DContext!Q: What is 98 plus 45?
A:
Target Completion !143
Figure G.44: Formatted dataset example for Arithmetic 2D+
Context!Q: What is 95 times 45?
A:
Target Completion !4275
Figure G.45: Formatted dataset example for Arithmetic 2Dx
Context!Q: What is 509 minus 488?
A:
Target Completion !21
Figure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?
A:
Target Completion !1053

Language Models are Few-Shot Learners

40+32 pages


Figure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?
A:
Target Completion !1053
Figure G.47: Formatted dataset example for Arithmetic 3D+
Context!Q: What is 6209 minus 3365?
A:
Target Completion !2844
Figure G.48: Formatted dataset example for Arithmetic 4D61
Context!Q: What is 9923 plus 617?
A:
Target Completion !10540
Figure G.49: Formatted dataset example for Arithmetic 4D+
Context!Q: What is 40649 minus 78746?
A:
Target Completion !-38097
Figure G.50: Formatted dataset example for Arithmetic 5D
Context!Q: What is 65360 plus 16204?
A:
Target Completion !81564
Figure G.51: Formatted dataset example for Arithmetic 5D+
62
H Results on All Tasks for All Model Sizes
Zero-Shot One-Shot Few-Shot
Name Metric SplitFine-tune
SOTA K Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B175B

Language Models are Few-Shot Learners

40+32 pages


Language Models are Few-Shot Learners

40+32 pages


63
Figure H.1: All results for all SuperGLUE tasks.
Figure H.2: Results for SAT task.
 Figure H.3: All results for all Winograd tasks.
64
Figure H.4: All results for all Arithmetic tasks.
Figure H.5: All results for all Cloze and Completion tasks.
65
Figure H.6: All results for all Common Sense Reasoning tasks.
Figure H.7: All results for all QA tasks.
Figure H.8: All results for all Reading Comprehension tasks.
Figure H.9: All results for all ANLI rounds.
66
Figure H.10: All results for all Scramble tasks.
Figure H.11: All results for all Translation tasks.
67
References
[ADG+16]Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul,
Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent.
InAdvances in neural information processing systems , pages 3981–3989, 2016.
[AI19] WeChat AI. Tr-mt (ensemble), December 2019.
[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In

Language Models are Few-Shot Learners

40+32 pages


[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.
[BBDIW20] Su Lin Blodgett, Solon Barocas, Hal Daum ´e III, and Hanna Wallach. Language (technology) is power:
A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050 , 2020.
[BCFL13] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from
question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language
processing , pages 1533–1544, 2013.
[BDD+09]Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The ﬁfth
PASCAL recognizing textual entailment challenge. 2009.
[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical

Language Models are Few-Shot Learners

40+32 pages


[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical
resource for sentiment analysis and opinion mining. In Lrec, volume 10, pages 2200–2204, 2010.
[BHDD+06]Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan
Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
[BHT+20]Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella
Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language.
arXiv preprint arXiv:2004.10151 , 2020.
[BLC13] Yoshua Bengio, Nicholas L ´eonard, and Aaron C. Courville. Estimating or propagating gradients through
stochastic neurons for conditional computation. Arxiv , 2013.
[BZB+19]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about

Language Models are Few-Shot Learners

40+32 pages


physical commonsense in natural language. arXiv preprint arXiv:1911.11641 , 2019.
[Car97] Rich Caruana. Multitask learning. Machine learning , 28(1), 1997.
[CB78] Susan Carey and Elsa Bartlett. Acquiring a single new word. Proceedings of the Stanford Child Language
Conference , 1978.
[CCE+18]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv ,
abs/1803.05457, 2018.
[CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse
transformers, 2019.
[CHI+18]Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke
Zettlemoyer. Quac : Question answering in context. Arxiv , 2018.
[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina

Language Models are Few-Shot Learners

40+32 pages


[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Toutanova. BoolQ: Exploring the surprising difﬁculty of natural yes/no questions. arXiv preprint
arXiv:1905.10044 , 2019.
[CLY+19]Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and
Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 ,
2019.
[Cra17] Kate Crawford. The trouble with bias. NIPS 2017 Keynote , 2017.
[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.
68
[DGM06] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment
challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classiﬁcation,

Language Models are Few-Shot Learners

40+32 pages


challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classiﬁcation,
and recognising textual entailment , pages 177–190. Springer, 2006.
[DGV+18]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal
transformers. Arxiv , 2018.
[DHKH14] Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heaﬁeld. Edinburgh’s phrase-based machine
translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation ,
pages 97–104, 2014.
[DL15] Andrew M. Dai and Quoc V . Le. Semi-supervised sequence learning. In Advances in neural information
processing systems , 2015.
[DMST19] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung
23. Data can be found at https://github.com/mcdm/CommitmentBank/.

Language Models are Few-Shot Learners

40+32 pages


23. Data can be found at https://github.com/mcdm/CommitmentBank/.
[DSC+16]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast
reinforcement learning via slow reinforcement learning. ArXiv , abs/1611.02779, 2016.
[DWD+19]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint
arXiv:1903.00161 , 2019.
[DYY+19]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V . Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a ﬁxed-length context. Arxiv , 2019.
[EOAG18] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale.
arXiv preprint arXiv:1808.09381 , 2018.

Language Models are Few-Shot Learners

40+32 pages


arXiv preprint arXiv:1808.09381 , 2018.
[FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of
deep networks. ArXiv , abs/1703.03400, 2017.
[Fyo00] Yaroslav Fyodorov. A natural logic inference system, 2000.
[GG19] Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases
in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862 , 2019.
[GLT+20]Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909 , 2020.
[GMDD07] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing
textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and
paraphrasing , pages 1–9. Association for Computational Linguistics, 2007.

Language Models are Few-Shot Learners

40+32 pages


paraphrasing , pages 1–9. Association for Computational Linguistics, 2007.
[Gra16] Alex Graves. Adaptive computation time for recurrent neural networks. Arxiv , 2016.
[GSL+18]Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A
Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324 , 2018.
[GSR19] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 1906.04043 , 2019.
[GWC+18]Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. Meta-learning for low-resource
neural machine translation. arXiv preprint arXiv:1808.08437 , 2018.
[HB20] Daniel Hernandez and Tom Brown. Ai and efﬁciency, May 2020.
[HBFC19] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.
CoRR , abs/1904.09751, 2019.

Language Models are Few-Shot Learners

40+32 pages


CoRR , abs/1904.09751, 2019.
[HLW+20]Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song.
Pretrained transformers improve out of distribution robustness. arXiv preprint arXiv:2004.06100 , 2020.
69
[HNA+17]Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.
Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.
arXiv preprint arXiv:1712.00409 , 2017.
[HR18] Jeremy Howard and Sebastian Ruder. Universal language model ﬁne-tuning for text classiﬁcation. arXiv
preprint arXiv:1801.06146 , 2018.
[HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
preprint arXiv:1503.02531 , 2015.
[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent.

Language Models are Few-Shot Learners

40+32 pages


[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent.
InInternational Conference on Artiﬁcial Neural Networks , pages 87–94. Springer, 2001.
[HZJ+19]Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini,
Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual
evaluation. arXiv preprint arXiv:1911.03064 , 2019.
[IBGC+14]Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daum ´e III. A neural
network for factoid question answering over paragraphs. In Empirical Methods in Natural Language
Processing , 2014.
[IDCBE19] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of
generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650 , 2019.
[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly

Language Models are Few-Shot Learners

40+32 pages


[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly
supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.
[JN20] Zheng Junyuan and Gamma Lab NYC. Numeric transformer - albert, March 2020.
[JVS+16]Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits
of language modeling. arXiv preprint arXiv:1602.02410 , 2016.
[JYS+19]Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.
TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351 , 2019.
[JZC+19]Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on
conversational question answering. arXiv preprint arXiv:1909.10772 , 2019.

Language Models are Few-Shot Learners

40+32 pages


conversational question answering. arXiv preprint arXiv:1909.10772 , 2019.
[KCR+18]Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond
the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North
American Chapter of the Association for Computational Linguistics (NAACL) , 2018.
[KKS+20]Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.
Uniﬁedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700 , 2020.
[KMB20] Sarah E. Kreps, Miles McCain, and Miles Brundage. All the news that’s ﬁt to fabricate: Ai-generated
text as a tool of media misinformation, 2020.
[KMH+20]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott
Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.

Language Models are Few-Shot Learners

40+32 pages


Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[KPR+19]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redﬁeld, Michael Collins, Ankur Parikh, Chris Alberti,
Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova,
Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational
Linguistics , 2019.
[KR16] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. Arxiv , 2016.
[LB02] Edward Loper and Steven Bird. Nltk: The natural language toolkit, 2002.
[LC19] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint
arXiv:1901.07291 , 2019.
70

Language Models are Few-Shot Learners

40+32 pages


arXiv:1901.07291 , 2019.
70
[LCG+19]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint
arXiv:1909.11942 , 2019.
[LCH+20]Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao.
Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 , 2020.
[LDL19] Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable bert. arXiv preprint
arXiv:1905.07504 , 2019.
[LDM12] Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth
International Conference on the Principles of Knowledge Representation and Reasoning , 2012.
[LGG+20]Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and

Language Models are Few-Shot Learners

40+32 pages


Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint
arXiv:2001.08210 , 2020.
[LGH+15]Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation
learning using multi-task deep neural networks for semantic classiﬁcation and information retrieval. In
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies , 2015.
[LH17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101 , 2017.
[LHCG19a] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural
networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482 ,
2019.
[LHCG19b] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for
natural language understanding. arXiv preprint arXiv:1901.11504 , 2019.

Language Models are Few-Shot Learners

40+32 pages


natural language understanding. arXiv preprint arXiv:1901.11504 , 2019.
[Lin20] Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint
arXiv:2005.00955 , 2020.
[LLG+19]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.
[LM17] Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441 , 2017.
[LOG+19]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.
arXiv preprint arXiv:1907.11692 , 2019.

Language Models are Few-Shot Learners

40+32 pages


arXiv preprint arXiv:1907.11692 , 2019.
[LPP+20]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Kiela Douwe.
Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401 ,
2020.
[LSP+18]Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam
Shazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 , 2018.
[LWS+20]Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez.
Train large, then compress: Rethinking model size for efﬁcient training and inference of transformers,
2020.
[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading

Language Models are Few-Shot Learners

40+32 pages


[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading
comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 , 2017.
[LYN+20]Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy
Lin. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380 , 2020.
[Mac92] David. MacKay. Information-based objective functions for active data selection. Neural Computation ,
1992.
71
[MBXS17] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pages 6294–6305,
2017.
[MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efﬁcient estimation of word representations
in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Language Models are Few-Shot Learners

40+32 pages


in vector space. arXiv preprint arXiv:1301.3781 , 2013.
[MCH+16]Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende,
Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of
commonsense stories. arXiv preprint arXiv:1604.01696 , 2016.
[MCKS18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity?
a new dataset for open book question answering. ArXiv , abs/1809.02789, 2018.
[MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of
large-batch training, 2018.
[MKM+94]Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson,
Karen Katz, and Britta Schasberger. The penn treebank: annotating predicate argument structure.
InProceedings of the workshop on Human Language Technology , pages 114–119. Association for
Computational Linguistics, 1994.

Language Models are Few-Shot Learners

40+32 pages


InProceedings of the workshop on Human Language Technology , pages 114–119. Association for
Computational Linguistics, 1994.
[MKXS18] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language
decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 , 2018.
[MPL19] R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic
heuristics in natural language inference. arXiv preprint arXiv:1902.01007 , 2019.
[MWZ+18]Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson,
Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting, 2018.
[NBR20] Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained
language models. arXiv preprint arXiv:2004.09456 , 2020.
[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments.

Language Models are Few-Shot Learners

40+32 pages


[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments.
arXiv preprint arXiv:1907.07355 , 2019.
[Nor09] Peter Norvig. Natural language corpus data, 2009.
[NvNvdG19] Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational: Man is to doctor
as woman is to doctor. arXiv preprint arXiv:1905.09866 , 2019.
[NWD+19]Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial
nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 , 2019.
[oR16] University of Regensburg. Fascha, 2016.
[PCC18] Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating
context-sensitive representations. arXiv preprint arXiv:1808.09121 , 2018.
[PFB18] Jason Phang, Thibault F ´evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary

Language Models are Few-Shot Learners

40+32 pages


[PFB18] Jason Phang, Thibault F ´evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary
training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018.
[PHR+18]Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and
Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation
evaluation. In Proceedings of EMNLP , 2018.
[PKL+16]Denis Paperno, Germ ´an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro
Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern ´andez. The lambada dataset: Word prediction
requiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016.
[PNZtY18] Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih. Dissecting contextual word
embeddings: Architecture and representation, 2018.

Language Models are Few-Shot Learners

40+32 pages


embeddings: Architecture and representation, 2018.
[Pos18] Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771 , 2018.
72
[PSM14] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language
processing (EMNLP) , 2014.
[QIA20] QIANXIN. Sa-net on albert (ensemble), April 2020.
[QMZH19] Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language
models with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801 , 2019.
[RBG11] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An
evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series , 2011.
[RCM19] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering
challenge. Transactions of the Association for Computational Linguistics , 7:249–266, 2019.

Language Models are Few-Shot Learners

40+32 pages


challenge. Transactions of the Association for Computational Linguistics , 7:249–266, 2019.
[RCP+17]Scott Reed, Yutian Chen, Thomas Paine, A ¨aron van den Oord, SM Eslami, Danilo Rezende, Oriol
Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn
distributions. arXiv preprint arXiv:1710.10304 , 2017.
[RJL18] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for
squad. arXiv preprint arXiv:1806.03822 , 2018.
[RL16] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR 2017 (oral) ,
2016.
[RLL+19]Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. NumNet: Machine reading comprehension
with numerical reasoning. In Proceedings of EMNLP , 2019.
[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in

Language Models are Few-Shot Learners

40+32 pages


[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in
coreference resolution. arXiv preprint arXiv:1804.09301 , 2018.
[RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding
by generative pre-training, 2018.
[Ros12] R.S. Ross. Guide for conducting risk assessments. NIST Special Publication , 2012.
[RRBS19] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of
the generalization error across scales, 2019.
[RRS20] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters
of a language model? arXiv preprint arXiv:2002.08910 , 2020.
[RSR+19]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer, 2019.

Language Models are Few-Shot Learners

40+32 pages


Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer, 2019.
[RWC+19]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners, 2019.
[SBBC19] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial
winograd schema challenge at scale, 2019.
[SBC+19]Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Jeff Wu, Alec Radford,
Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris
McGufﬁe, and Jasmine Wang. Release strategies and the social impacts of language models, 2019.
[SCNP19] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a
babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019.

Language Models are Few-Shot Learners

40+32 pages


babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019.
[SDCW19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of
BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.
[SDSE19] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. CoRR , abs/1907.10597, 2019.
[SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with
monolingual data. arXiv preprint arXiv:1511.06709 , 2015.
73
[SMM+17]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff
Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint
arXiv:1701.06538 , 2017.

Language Models are Few-Shot Learners

40+32 pages


arXiv:1701.06538 , 2017.
[SPP+19]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019.
[SS20] Timo Schick and Hinrich Sch ¨utze. Exploiting cloze questions for few-shot text classiﬁcation and natural
language inference. arXiv preprint arXiv:2001.07676 , 2020.
[STQ+19]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence
pre-training for language generation. arXiv preprint arXiv:1905.02450 , 2019.
[TFR+17]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain
randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ
international conference on intelligent robots and systems (IROS) , pages 23–30. IEEE, 2017.

Language Models are Few-Shot Learners

40+32 pages


international conference on intelligent robots and systems (IROS) , pages 23–30. IEEE, 2017.
[TL05] Peter D. Turney and Michael L. Littman. Corpus-based learning of analogies and semantic relations.
CoRR , abs/cs/0508103, 2005.
[TL18] Trieu H. Trinh and Quoc V . Le. A simple method for commonsense reasoning. arXiv preprint
arXiv:1806.02847 , 2018.
[TLBS03] Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. Combining independent
modules to solve multiple-choice synonym and analogy problems. CoRR , cs.CL/0309035, 2003.
[Tur20] Project Turing. Microsoft research blog, Feb 2020.
[VBL+16]Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching Networks for One
Shot Learning. In Advances in neural information processing systems , pages 3630–3638, 2016.
[VSP+17]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz

Language Models are Few-Shot Learners

40+32 pages


Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing
systems , 2017.
[WPN+19]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer
Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems , pages 3261–3275, 2019.
[WXH+18]Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Multi-agent
dual learning. ICLR 2019 , 2018.
[XDH+19]Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V . Le. Unsupervised data
augmentation for consistency training, 2019.
[YdC+19]Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski,
Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating

Language Models are Few-Shot Learners

40+32 pages


Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating
general linguistic intelligence. arXiv preprint arXiv:1901.11373 , 2019.
[YDY+19]Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V . Le. XLNet:
Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 ,
2019.
[ZHB+19]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine
really ﬁnish your sentence? arXiv preprint arXiv:1905.07830 , 2019.
[ZHR+19]Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin
Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616 , 2019.
[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.

Language Models are Few-Shot Learners

40+32 pages


[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.
ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv
preprint arXiv:1810.12885 , 2018.
[ZSW+19a] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul
Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
74
[ZSW+19b] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv , abs/1909.08593,
2019.
75

Language Models are Few-Shot Learners

40+32 pages


Uncertainty-Aware Weakly Supervised Action
Detection from Untrimmed Videos
Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia Schmid
Google Research
faarnab, chensun, anagrani, cordelias g@google.com
Abstract. Despite the recent advances in video classication, progress
in spatio-temporal action recognition has lagged behind. A major contributing factor has been the prohibitive cost of annotating videos frameby-frame. In this paper, we present a spatio-temporal action recognition
model that is trained with only video-level labels, which are signicantly
easier to annotate. Our method leverages per-frame person detectors
which have been trained on large image datasets within a Multiple Instance Learning framework. We show how we can apply our method in
cases where the standard Multiple Instance Learning assumption, that
each bag contains at least one instance with the specied label, is invalid using a novel probabilistic variant of MIL where we estimate the
uncertainty of each prediction. Furthermore, we report the rst weaklysupervised results on the AVA dataset and state-of-the-art results among
weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video

Summary:
Despite the recent advances in video classification, progress in
spatio-temporal action recognition has lagged behind. A major contributing
factor has been the prohibitive cost of annotating videos frame-by-frame. In
this paper, we present a spatio-temporal action recognition model that is
trained with only video-level labels, which are significantly easier to
annotate. Our method leverages per-frame person detectors which have been
trained on large image datasets within a Multiple Instance Learning framework.
We show how we can apply our method in cases where the standard Multiple
Instance Learning assumption, that each bag contains at least one instance with
the specified label, is invalid using a novel probabilistic variant of MIL
where we estimate the uncertainty of each prediction. Furthermore, we report
the first weakly-supervised results on the AVA dataset and state-of-the-art
results among weakly-supervised methods on UCF101-24.

Anurag Arnab
Chen Sun
Arsha Nagrani
Cordelia Schmid

ECCV 2020


weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video
understanding, mulitple instance learning
1 Introduction
Video classication has witnessed great advances recently due to large datasets
such as Kinetics [ 20] and Moments in Time [ 30] which have enabled training
of specialised neural network architectures for video [ 5,9]. However, progress in
other video understanding tasks, such as spatio-temporal action detection, has
lagged behind in comparison. There are fewer datasets for action recognition,
which are also signicantly smaller than their video-classication counterparts. A
reason for this is the exorbitant cost of annotating videos with spatio-temporal
labels { each frame of an action has to be manually labelled with a bounding
box. Moreover, annotating temporal boundaries of actions is not only arduous,
but often ambiguous with annotators failing to reach consensus about the start
and end times of an action [6,39].
In this paper, we propose a method to train spatio-temporal action detectors
using only weak, video-level annotations as shown in Fig. 1. To achieve this, we
leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations

ECCV 2020


leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations
in appearance, scene and pose. We adopt a Multiple Instance Learning (MIL)arXiv:2007.10703v1  [cs.CV]  21 Jul 2020
2 A. Arnab et al.
Training dataPrediction
Labels:
sit, talk to (people), watch (a person), carry/hold (an object), listen to (a person)
 sit, carry/hold (an object), listen to (a person), talk to (people), watch (a person)
sit, talk to (people), watch (a person)
sit, listen to (a person), watch (a person)
time
Fig. 1. We propose a method to train a spatio-temporal action detector using only
weak, video-level labels on challenging, real-world datasets. Note that the video-level
labels that we have may apply to multiple people in the video, and that these labels
may only be active for an unannotated time interval of the input clip.
framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of

ECCV 2020