our entire dataset in around seven hours (including time to tokenize), utilizing an estimated 1,500 cores.
As there are many confounding factors at play when determining whether dataset contamination has
contributedtoevaluationperformance(mostlystemmingfromthefactthat"clean"and"dirty"subsetsdo
not necessarily well-estimate the population distribution), we make the following assumption: In the event
of dataset contamination contributing to evaluation performance, we expect both the "cleanest" examples to
haveanoverall worseaveragescorethantheircomplement,andthe"dirtiest"samplestohaveanoverall better
average score than their complement. It is insuﬃcient evidence for contamination if only one of these were
true. To this end, we deﬁne four (non-disjoint) subset types as follows:
•“Clean”samples, with less than 20% token contamination,
•“Not clean” samples, with greater than (or equal to) 20% token contamination,
•“Not dirty” samples, with less than 80% token contamination,
•“Dirty”samples, with greater than (or equal to) 80% token contamination.
Thereisanadditionalconfoundingfactorthatweattempttoaddressdirectly. Withthegivendeﬁnitionof


Thereisanadditionalconfoundingfactorthatweattempttoaddressdirectly. Withthegivendeﬁnitionof
contamination(aswellasotherdeﬁnitionsmentionedintheliterature),thereisapossibilitythatasample
75
Dataset Model Subset Type Avg. Contam. % n X nZn
HellaSwag ( L= 40)70BClean 0 7391 80.0 82.5 -5.73
Not Clean 67.5 2651 89.5 82.4 9.56
Not Dirty 11.5 9194 81.6 82.5 -2.27
Dirty 86.1 848 92.2 82.5 7.42
7BClean 0 7391 70.5 73.3 -5.46
Not Clean 67.5 2651 81.3 73.4 9.17
Not Dirty 11.5 9194 72.4 73.4 -2.06
Dirty 86.1 848 83.7 73.3 6.84
MMLU-Humanities ( L= 50)70BClean 0.05 3996 62.2 65.3 -4.08
Not Clean 85.12 709 82.7 65.3 9.71
Not Dirty 2.73 4185 62.7 65.3 -3.50

Llama 2: Open Foundation and Fine-Tuned Chat Models

Summary:
In this work, we develop and release Llama 2, a collection of pretrained and
fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70
billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for
dialogue use cases. Our models outperform open-source chat models on most
benchmarks we tested, and based on our human evaluations for helpfulness and
safety, may be a suitable substitute for closed-source models. We provide a
detailed description of our approach to fine-tuning and safety improvements of
Llama 2-Chat in order to enable the community to build on our work and
contribute to the responsible development of LLMs.

Hugo Touvron
Louis Martin
Kevin Stone
Peter Albert
Amjad Almahairi
Yasmine Babaei
Nikolay Bashlykov
Soumya Batra
Prajjwal Bhargava
Shruti Bhosale
Dan Bikel
Lukas Blecher
Cristian Canton Ferrer
Moya Chen
Guillem Cucurull
David Esiobu
Jude Fernandes
Jeremy Fu
Wenyin Fu
Brian Fuller
Cynthia Gao
Vedanuj Goswami
Naman Goyal
Anthony Hartshorn
Saghar Hosseini
Rui Hou
Hakan Inan
Marcin Kardas
Viktor Kerkez
Madian Khabsa
Isabel Kloumann
Artem Korenev
Punit Singh Koura
Marie-Anne Lachaux
Thibaut Lavril
Jenya Lee
Diana Liskovich
Yinghai Lu
Yuning Mao
Xavier Martinet
Todor Mihaylov
Pushkar Mishra
Igor Molybog
Yixin Nie
Andrew Poulton
Jeremy Reizenstein
Rashi Rungta
Kalyan Saladi
Alan Schelten
Ruan Silva
Eric Michael Smith
Ranjan Subramanian
Xiaoqing Ellen Tan
Binh Tang
Ross Taylor
Adina Williams
Jian Xiang Kuan
Puxin Xu
Zheng Yan
Iliyan Zarov
Yuchen Zhang
Angela Fan
Melanie Kambadur
Sharan Narang
Aurelien Rodriguez
Robert Stojnic
Sergey Edunov
Thomas Scialom

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints


Not Clean 85.12 709 82.7 65.3 9.71
Not Dirty 2.73 4185 62.7 65.3 -3.50
Dirty 94.5 520 85.8 65.3 9.80
7BClean 0.05 3996 40.8 42.9 -2.75
Not Clean 85.2 709 54.9 42.8 6.50
Not Dirty 2.73 4185 41.1 42.9 -2.25
Dirty 94.5 520 56.9 42.8 6.49
MMLU-Overall ( L= 50) 70BClean 0.02 11862 68.0 68.9 -2.00
Not Clean 84.7 2180 73.5 68.9 4.64
Not Dirty 3.18 12506 67.7 68.9 -2.75
Dirty 94.4 1536 78.2 68.9 7.87
Table 51: Contamination analysis results for aﬀected datasets. No other evaluation datasets had suﬃcient
evidence to be considered aﬀected by contamination. Avg. Contam. % denotes the average per-sample
contamination percentage for the given subset type. Models sizes refer to pretrained-only models
may appear contaminated, by virtue of many tokens appearing in matched sequences found in the training


contamination percentage for the given subset type. Models sizes refer to pretrained-only models
may appear contaminated, by virtue of many tokens appearing in matched sequences found in the training
data. However, the matched sequences might be highly fragmented across the training data, in which case it
is very unlikely the model saw the correctly-assembled contaminated sequences during training. To reduce
the chance of this phenomenon, we repeat our analysis with minimum match length L2f10;20;30;40;50g.
Sinceinthelimitof L!1everysamplefallsintoboththe"clean"and"notdirty"(thereisnocontamination),
we report the largest Lfor each dataset that appeared to beneﬁt from contamination to strike a balance
between fragmentation and overall contamination.
Foreachdatasetandeachoftheabovesamplesubsettypes,wecomputeboththemean Xoftheperformance
metricXand the statistic Zn=(Xn)
n, wherenis the size of the sample subset type, and nand2
nare the
mean and variance of the sampling distribution of the performance metric for samples of size n, respectively.
BytheCentralLimitTheorem, Zntendstowardsastandardnormaldistributionandsoweconsiderthere


BytheCentralLimitTheorem, Zntendstowardsastandardnormaldistributionandsoweconsiderthere
issuﬃcientevidencetosuggestcontaminationhasaﬀectedevaluationperformanceonadatasetifallfour
sample subsets have jZnj>2.
ResultsforthisanalysiscanbeseeninTable51. WeobservethatonlyHellaSwagandMMLU-Humanities
appear to have been boosted due to contamination in the training data, with the 70B model appearing to
have gained a greater beneﬁt than the 7B model, as one might expect. Furthermore, the impact of this eﬀect
onMMLU-HumanitiesappearstocauseabeneﬁtforMMLU-Overallforthe70Bmodel,albeitwithonlya
small delta (-0.9) between the "clean" subset performance and the sampling mean. No other dataset (for any
choice ofL) appears to have beneﬁtted from dataset contamination, and we omit results from these datasets
for conciseness.
76
A.7 Model Card
Table 52 presents a model card (Mitchell et al., 2018; Anil et al., 2023) that summarizes details of the models.
Model Details
Model Developers Meta AI


Table 52 presents a model card (Mitchell et al., 2018; Anil et al., 2023) that summarizes details of the models.
Model Details
Model Developers Meta AI
Variations L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle comes in a range of parameter sizes—7B, 13B, and 70B—as well as
pretrained and ﬁne-tuned variations.
Input Models input text only.
Output Models generate text only.
Model Architecture L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle isanauto-regressivelanguagemodelthatusesanoptimizedtransformer
architecture. Thetunedversionsusesupervisedﬁne-tuning(SFT)andreinforcementlearning withhuman feedback(RLHF)to aligntohuman preferencesfor
helpfulness and safety.
Model Dates L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle was trained between January 2023 and July 2023.
Status This is a static model trained on an oﬄine dataset. Future versions of the tuned
models will be released as we improve model safety with community feedback.
License A custom commercial license is available at: ai.meta.com/resources/


models will be released as we improve model safety with community feedback.
License A custom commercial license is available at: ai.meta.com/resources/
models-and-libraries/llama-downloads/
Where to send commentsInstructions on how to provide feedback or comments on the model can be
found in the model README, or by opening an issue in the GitHub repository
(https://github.com/facebookresearch/llama/ ).
Intended Use
Intended Use Cases L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle is intended for commercial and research use in English. Tuned models
are intended for assistant-like chat, whereas pretrained models can be adapted
for a variety of natural language generation tasks.
Out-of-Scope Uses Use in any manner that violates applicable laws or regulations (including trade
compliancelaws). UseinlanguagesotherthanEnglish. Useinanyotherway
that is prohibited by the Acceptable Use Policy and Licensing Agreement for
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle.
Hardware and Software (Section 2.2)
Training Factors We usedcustomtraininglibraries, Meta’sResearchSuperCluster, andproductionclustersforpretraining. Fine-tuning,annotation,andevaluationwerealso


Training Factors We usedcustomtraininglibraries, Meta’sResearchSuperCluster, andproductionclustersforpretraining. Fine-tuning,annotation,andevaluationwerealso
performed on third-party cloud compute.
Carbon Footprint Pretraining utilized a cumulative 3.3M GPU hours of computation on hardware
of type A100-80GB (TDP of 350-400W). Estimated total emissions were 539
tCO 2eq, 100% of which were oﬀset by Meta’s sustainability program.
Training Data (Sections 2.1 and 3)
Overview L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle was pretrained on 2 trillion tokens of data from publicly available
sources. The ﬁne-tuning data includes publicly available instruction datasets, as
wellasoveronemillionnewhuman-annotatedexamples. Neitherthepretraining
nor the ﬁne-tuning datasets include Meta user data.
Data Freshness The pretraining data has a cutoﬀ of September 2022, but some tuning data is
more recent, up to July 2023.
Evaluation Results
See evaluations for pretraining (Section 2); ﬁne-tuning (Section 3); and safety (Section 4).


Evaluation Results
See evaluations for pretraining (Section 2); ﬁne-tuning (Section 3); and safety (Section 4).
Ethical Considerations and Limitations (Section 5.2)
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle is a new technology that carries risks with use. Testing conducted to date has been in
English, and has notcovered, nor could it coverall scenarios. For these reasons, aswith all LLMs,
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle’s potential outputs cannot be predicted in advance, and the model may in some instances
produceinaccurateorobjectionableresponsestouserprompts. Therefore,beforedeployingany
applications of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle, developers should perform safety testing and tuning tailored to their
speciﬁc applications of the model. Please see the Responsible Use Guide available available at
https://ai.meta.com/llama/responsible-user-guide
Table 52: Model card for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle .
77