GQA: Training Generalized Multi-Query Transformer Models from
Multi-Head Checkpoints
Joshua Ainslie, James Lee-Thorp, Michiel de Jongyy
Yury Zemlyanskiy ,Federico Lebrón ,Sumit Sanghai
Google Research
Abstract
Multi-query attention (MQA), which only
uses a single key-value head, drastically
speeds up decoder inference. However, MQA
can lead to quality degradation, and moreover
it may not be desirable to train a separate
model just for faster inference. We (1) propose
a recipe for uptraining existing multi-head language model checkpoints into models with
MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than
one, less than number of query heads) number
of key-value heads. We show that uptrained
GQA achieves quality close to multi-head attention with comparable speed to MQA.
1 Introduction
Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory
bandwidth overhead from loading decoder weights
and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong
et al., 2022). The memory bandwidth from loading

Joshua Ainslie
James Lee-Thorp
Michiel de Jong
Yury Zemlyanskiy
Federico Lebrón
Sumit Sanghai

Accelerating large language model decoding with speculative sampling


and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong
et al., 2022). The memory bandwidth from loading
keys and values can be sharply reduced through
multi-query attention (Shazeer, 2019), which uses
multiple query heads but single key and value
heads.
However, multi-query attention ( MQA ) can lead
to quality degradation and training instability, and
it may not be feasible to train separate models
optimized for quality and inference. Moreover,
while some language models already use multiquery attention, such as PaLM (Chowdhery et al.,
2022), many do not, including publicly available
language models such as T5 (Raffel et al., 2020)
and LLaMA (Touvron et al., 2023).
This work contains two contributions for faster
inference with large language models. First, we
Equal contribution.
yUniversity of Southern California. Work done at Google
Research.show that language model checkpoints with multihead attention ( MHA ) can be uptrained (Komatsuzaki et al., 2022) to use MQA with a small fraction of original training compute. This presents a
cost-effective method to obtain fast multi-query as
well as high-quality MHA checkpoints.
Second, we propose grouped-query attention


cost-effective method to obtain fast multi-query as
well as high-quality MHA checkpoints.
Second, we propose grouped-query attention
(GQA ), an interpolation between multi-head and
multi-query attention with single key and value
heads per subgroup of query heads . We show that
uptrained GQA achieves quality close to multihead attention while being almost as fast as multiquery attention.
2 Method
2.1 Uptraining
Generating a multi-query model from a multi-head
model takes place in two steps: ﬁrst, converting the
checkpoint, and second, additional pre-training to
allow the model to adapt to its new structure. Figure 1 shows the process for converting a multi-head
checkpoint into a multi-query checkpoint. The projection matrices for key and value heads are mean
pooled into single projection matrices, which we
ﬁnd works better than selecting a single key and
value head or randomly initializing new key and
value heads from scratch.
Figure 1: Overview of conversion from multi-head to
multi-query attention. Key and value projection matrices from all heads are mean pooled into a single head.
The converted checkpoint is then pre-trained forarXiv:2305.13245v1  [cs.CL]  22 May 2023


The converted checkpoint is then pre-trained forarXiv:2305.13245v1  [cs.CL]  22 May 2023
Figure 2: Overview of grouped-query method. Multi-head attention has H query, key, and value heads. Multi-query
attention shares single key and value heads across all query heads. Grouped-query attention instead shares single
key and value heads for each group of query heads, interpolating between multi-head and multi-query attention.
a small proportion of its original training steps
on the same pre-training recipe.
2.2 Grouped-query attention
Grouped-query attention divides query heads into
Ggroups , each of which shares a single key head
and value head. GQA -Grefers to grouped-query
withGgroups. GQA -1, with a single group and
therefore single key and value head, is equivalent to
MQA , while GQA -H, with groups equal to number
of heads, is equivalent to MHA . Figure 2 shows a
comparison of grouped-query attention and multihead/multi-query attention. When converting a
multi-head checkpoint to a GQA checkpoint, we
construct each group key and value head by meanpooling all the original heads within that group.
An intermediate number of groups leads to an


construct each group key and value head by meanpooling all the original heads within that group.
An intermediate number of groups leads to an
interpolated model that is higher quality than MQA
but faster than MHA , and, as we will show, represents a favorable trade-off. Going from MHA
toMQA reducesHkey and value heads to a single key and value head, reducing the size of the
key-value cache and therefore amount of data that
needs to be loaded by a factor of H. However,
larger models generally scale the number of heads,
such that multi-query attention represents a more
aggressive cut in both memory bandwidth and capacity. GQA lets us keep the same proportional
decrease in bandwidth and capacity as model size
increases.
Moreover, larger models suffer relatively less
from memory bandwidth overhead from attention,
as the KV-cache scales with model dimension
while model FLOPs and parameters scale with the
square of model dimension. Finally, standard sharding for large models replicates the single key and
value head by the number of model partitions (Popeet al., 2022); GQA removes the waste from such
partitioning. Therefore, we expect GQA to present
a particularly good trade-off for larger models.
3 Experiments
3.1 Experimental setup
Conﬁgurations All models are based on the


a particularly good trade-off for larger models.
3 Experiments
3.1 Experimental setup
Conﬁgurations All models are based on the
T5.1.1 architecture (Raffel et al., 2020), implemented with JAX (Bradbury et al., 2018),
Flax (Heek et al., 2020), and Flaxformer1. For
our main experiments we consider T5 Large and
XXL with multi-head attention, as well as uptrained
versions of T5 XXL with multi-query and groupedquery attention. We apply MQA andGQA to
decoder self-attention and cross-attention, but not
encoder self-attention.
Uptraining Uptrained models are initialized
from public T5.1.1 checkpoints. The key and value
heads are mean-pooled to the appropriate MQA or
GQA structure, and then pre-trained for a further
proportion of original pre-training steps with the
original pre-training setup (Raffel et al., 2020).
Data We evaluate on summarization datasets
CNN/Daily Mail (Nallapati et al., 2016), arXiv
and PubMed (Cohan et al., 2018), MediaSum (Zhu
et al., 2021), and Multi-News (Fabbri et al., 2019);


and PubMed (Cohan et al., 2018), MediaSum (Zhu
et al., 2021), and Multi-News (Fabbri et al., 2019);
translation dataset WMT 2014 English-to-German;
and question answering dataset TriviaQA (Joshi
et al., 2017). We do not evaluate on popular classiﬁcation benchmarks such as GLUE (Wang et al.,
2019) as autoregressive inference is less applicable
for those tasks.
1https://github.com/google/flaxformer
Model Tinfer Average CNN arXiv PubMed MediaSum MultiNews WMT TriviaQA
s R1 R1 R1 R1 R1 BLEU F1
MHA-Large 0.37 46.0 42.9 44.6 46.2 35.5 46.6 27.7 78.2
MHA-XXL 1.51 47.2 43.8 45.6 47.5 36.4 46.9 28.4 81.9
MQA-XXL 0.24 46.5 43.0 45.0 47.2 36.1 46.5 28.5 81.3
GQA-8-XXL 0.28 47.1 43.5 45.4 47.7 36.3 47.2 28.4 81.6


GQA-8-XXL 0.28 47.1 43.5 45.4 47.7 36.3 47.2 28.4 81.6
Table 1: Inference time and average dev set performance comparison of T5 Large and XXL models with multihead attention, and 5% uptrained T5-XXL models with multi-query and grouped-query attention on summarization
datasets CNN/Daily Mail, arXiv, PubMed, MediaSum, and MultiNews, translation dataset WMT, and questionanswering dataset TriviaQA.
Fine-tuning For ﬁne-tuning, we use a constant
learning rate of 0.001, batch size 128, and dropout
rate 0.1 for all tasks. CNN/Daily Mail and WMT
use input length of 512 and output length 256.
Other summarization datasets use input length
2048 and output length 512. Finally, TriviaQA
uses input length 2048 and output length 32. We
train until convergence and select the checkpoint
with the highest dev performance. We use greedy
decoding for inference.
Timing We report time per sample per TPUv4
chip, as measured by xprof (Google, 2020). For
timing experiments we use 8 TPUs with the largest


Timing We report time per sample per TPUv4
chip, as measured by xprof (Google, 2020). For
timing experiments we use 8 TPUs with the largest
batch size that ﬁts up to 32 per TPU, and parallelization optimized separately for each model.
0 0:5 1 1:54646:547
MHA-LargeMHA-XXL
MQA-XXLGQA-XXL
Time per sample (ms)Performance
Figure 3: Uptrained MQA yields a favorable tradeoff compared to MHA with higher quality and
faster speed than MHA-Large, and GQA achieves
even better performance with similar speed gains
and comparable quality to MHA-XXL. Average performance on all tasks as a function of average inference
time per sample for T5-Large and T5-XXL with multihead attention, and 5% uptrained T5-XXL with MQA
and GQA-8 attention.3.2 Main results
Figure 3 shows average performance over all
datasets as a function of average inference time
forMHA T5-Large and T5-XXL, and uptrained
MQA andGQA -8XXL models with uptraining


forMHA T5-Large and T5-XXL, and uptrained
MQA andGQA -8XXL models with uptraining
proportion= 0:05. We see that a larger uptrained MQA model provides a favorable tradeoff relative to MHA models, with higher quality
and faster inference than MHA -Large. Moreover,
GQA achieves signiﬁcant additional quality gains,
achieving performance close to MHA -XXL with
speed close to MQA . Table 1 contains full results
for all datasets.
3.3 Ablations
This section presents experiments to investigate
the effect of different modeling choices. We evaluate performance on a representive subsample of
tasks: CNN/Daily Mail, (short-form summarization), MultiNews (long-form summarization), and
TriviaQA (question-answering).
54:454:654:855 55:255:455:6Mean
First
Random
Figure 4: Performance comparison of different checkpoint conversion methods for T5-Large uptrained to
MQA with proportion = 0:05. ‘Mean’ mean-pools
key and value heads, ‘First’ selects the ﬁrst head and
‘Random’ initializes heads from scratch.
Checkpoint conversion Figure 4 compares the


key and value heads, ‘First’ selects the ﬁrst head and
‘Random’ initializes heads from scratch.
Checkpoint conversion Figure 4 compares the
performance of different methods for checkpoint
conversion. Mean pooling appears to work best,
followed by selecting a single head and then random initialization. Intuitively, results are ordered
by the degree to which information is preserved
from the pre-trained model.
Uptraining steps Figure 5 shows how performance varies with uptraining proportion for T5
XXL with MQA andGQA . First, we note that
GQA already achieves reasonable performance after conversion while MQA requires uptraining to
be useful. Both MQA andGQA gain from 5%
uptraining with diminishing returns from 10%.
0 0:02 0:04 0:06 0:08 0:154555657
Uptraining proportion PerformanceMHA
GQA
MQA
Figure 5: Performance as a function of uptraining proportion for T5 XXL models with MQA and GQA-8.
Number of groups Figure 6 demonstrates the
effect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in


effect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in
key-value size is sharper due to the increased number of heads. As a result, increasing the number
of groups from MQA only results in modest slowdowns initially, with increasing cost as we move
closer to MHA . We selected 8 groups as a favorable middle ground.
4 Related Work
This work is focused on achieving a better tradeoff between decoder quality and inference time
through reducing the memory bandwidth overhead (Williams et al., 2009) from loading keys and
values. Shazeer (2019) ﬁrst proposed reducing this
overhead through multi-query attention. Follow-up
work showed that multi-query attention is especially helpful for long inputs (Pope et al., 2022;
de Jong et al., 2022).
A number of other methods have been proposed
to reduce memory bandwidth overhead from keys
and values, as well as parameters. Flash atten-1 4 8 16 32 6412
GQA groupsTime per sample (s)MHA
GQA
MQA
Figure 6: Time per sample for GQA-XXL as a function of the number of GQA groups with input length
2048 and output length 512. Going from 1 (MQA) to


2048 and output length 512. Going from 1 (MQA) to
8 groups adds modest inference overhead, with increasing cost to adding more groups.
tion (Dao et al., 2022) structures the attention computation to avoid materializing the quadratic attention scores, reducing memory and speeding up
training. Quantization (Dettmers et al., 2022; Zeng
et al., 2022; Frantar et al., 2022) reduces the size of
weights and activations, including keys and values,
by lowering precision. Model distillation (Hinton et al., 2015; Gou et al., 2021) instead reduces
model size at a given precision, using data generated from the larger model to ﬁnetune the smaller
model. Layer-sparse cross-attention (de Jong et al.,
2022) eliminates most cross-attention layers which
make up the primary expense for longer inputs.
Speculative sampling (Chen et al., 2023; Leviathan
et al., 2022; Kim et al., 2023) ameliorates the memory bandwidth bottleneck by proposing multiple
tokens with a smaller model which are then scored
in parallel by a larger model.
Finally, the uptraining procedure we propose
is inspired by Komatsuzaki et al. (2022), which
uptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.
5 Conclusion


is inspired by Komatsuzaki et al. (2022), which
uptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.
5 Conclusion
Language models are expensive for inference primarily due to the memory bandwidth overhead
from loading keys and values. Multi-query attention reduces this overhead at the cost of decreased
model capacity and quality. We propose to convert
multi-head attention models to multi-query models
with a small fraction of original pre-training compute. Moreover, we introduce grouped-query attention, an interpolation of multi-query and multi-head
attention that achieves quality close to multi-head
at comparable speed to multi-query attention.
Acknowlegements
We thank Santiago Ontañón, Afroz Mohiuddin,
William Cohen and others at Google Research for
insightful advice and discussion.
References
James Bradbury, Roy Frostig, Peter Hawkins,
Matthew James Johnson, Chris Leary, Dougal
Maclaurin, George Necula, Adam Paszke, Jake
VanderPlas, Skye Wanderman-Milne, and Qiao
Zhang. 2018. JAX: composable transformations of
Python+NumPy programs.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and
John Jumper. 2023. Accelerating large language


Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and
John Jumper. 2023. Accelerating large language
model decoding with speculative sampling. CoRR ,
abs/2302.01318.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts,
Paul Barham, Hyung Won Chung, Charles Sutton,
Sebastian Gehrmann, Parker Schuh, Kensen Shi,
Sasha Tsvyashchenko, Joshua Maynez, Abhishek
Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben
Hutchinson, Reiner Pope, James Bradbury, Jacob
Austin, Michael Isard, Guy Gur-Ari, Pengcheng
Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier
Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan,
Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,


Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,
Ryan Sepassi, David Dohan, Shivani Agrawal, Mark
Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz,
Erica Moreira, Rewon Child, Oleksandr Polozov,
Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta,
Jason Wei, Kathy Meier-Hellstern, Douglas Eck,
Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.
Palm: Scaling language modeling with pathways.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim,
Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention
model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 615–621,
New Orleans, Louisiana. Association for Computational Linguistics.
Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra,


New Orleans, Louisiana. Association for Computational Linguistics.
Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra,
and Christopher Ré. 2022. Flashattention: Fast and
memory-efﬁcient exact attention with io-awareness.
CoRR , abs/2205.14135.
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie,
Nicholas FitzGerald, Sumit Sanghai, Fei Sha, andWilliam Cohen. 2022. FiDO: Fusion-in-decoder optimized for stronger performance and faster inference. arXiv preprint arXiv:2212.08153 .
Tim Dettmers, Mike Lewis, Younes Belkada, and
Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. CoRR ,
abs/2208.07339.
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li,
and Dragomir R. Radev. 2019. Multi-news: A largescale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the
57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers , pages


57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July
28- August 2, 2019, Volume 1: Long Papers , pages
1074–1084. Association for Computational Linguistics.
Elias Frantar, Saleh Ashkboos, Torsten Hoeﬂer, and
Dan Alistarh. 2022. GPTQ: accurate post-training
quantization for generative pre-trained transformers.
CoRR , abs/2210.17323.
Google. 2020. Proﬁle your model with cloud tpu
tools. https://cloud.google.com/tpu/docs/
cloud-tpu-tools . Accessed: 2022-11-11.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and
Dacheng Tao. 2021. Knowledge distillation: A survey. Int. J. Comput. Vis. , 129(6):1789–1819.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner,
and Marc van Zee. 2020. Flax: A neural network
library and ecosystem for JAX.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.
2015. Distilling the knowledge in a neural network.


library and ecosystem for JAX.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.
2015. Distilling the knowledge in a neural network.
CoRR , abs/1503.02531.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke
Zettlemoyer. 2017. Triviaqa: A large scale distantly
supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of
the Association for Computational Linguistics , Vancouver, Canada. Association for Computational Linguistics.
Sehoon Kim, Karttikeya Mangalam, Jitendra Malik,
Michael W. Mahoney, Amir Gholami, and Kurt
Keutzer. 2023. Big little transformer decoder.
CoRR , abs/2302.07863.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp,
Carlos Riquelme Ruiz, Basil Mustafa, Joshua
Ainslie, Yi Tay, Mostafa Dehghani, and Neil
Houlsby. 2022. Sparse upcycling: Training mixtureof-experts from dense checkpoints.
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192.


Yaniv Leviathan, Matan Kalman, and Yossi Matias.
2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos
Santos, Çaglar Gülçehre, and Bing Xiang. 2016.
Abstractive text summarization using sequence-tosequence rnns and beyond. In Proceedings of the
20th SIGNLL Conference on Computational Natural
Language Learning, CoNLL 2016, Berlin, Germany,
August 11-12, 2016 , pages 280–290. ACL.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery,
Jacob Devlin, James Bradbury, Anselm Levskaya,
Jonathan Heek, Kefan Xiao, Shivani Agrawal, and
Jeff Dean. 2022. Efﬁciently scaling transformer inference. arXiv preprint arXiv:2211.05102 .
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu. 2020. Exploring the limits


Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J. Liu. 2020. Exploring the limits
of transfer learning with a uniﬁed text-to-text transformer. J. Mach. Learn. Res. , 21:140:1–140:67.
Noam Shazeer. 2019. Fast transformer decoding:
One write-head is all you need. arXiv preprint
arXiv:1911.02150 .
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, Aurelien Rodriguez, Armand Joulin,
Edouard Grave, and Guillaume Lample. 2023.
Llama: Open and efﬁcient foundation language models.
Alex Wang, Amanpreet Singh, Julian Michael, Felix
Hill, Omer Levy, and Samuel R. Bowman. 2019.
GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th
International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,
2019 . OpenReview.net.


International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,
2019 . OpenReview.net.
Samuel Williams, Andrew Waterman, and David A.
Patterson. 2009. Rooﬂine: an insightful visual performance model for multicore architectures. Commun. ACM , 52(4):65–76.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,
Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,
Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan
Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng
Zhang, Yuxiao Dong, and Jie Tang. 2022. GLM130B: an open bilingual pre-trained model. CoRR ,
abs/2210.02414.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng.
2021. Mediasum: A large-scale media interview
dataset for dialogue summarization. In Proceedings
of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, NAACL-HLT 2021,
Online, June 6-11, 2021 , pages 5927–5934. Association for Computational Linguistics.


Human Language Technologies, NAACL-HLT 2021,
Online, June 6-11, 2021 , pages 5927–5934. Association for Computational Linguistics.
A Training Stability
We ﬁnd that multi-query attention can lead to training instability, in particular combined with long
input tasks. We trained multiple T5-Large models with multi-query attention from scratch. In
each case, pre-training suffered from frequent loss
spikes and the ﬁnal models diverged immediately
on long-input tasks. Uptrained multi-query attention models are more stable but still display high
variance, so for multi-query models on unstable
tasks we report average performance over multiple
runs. Uptrained grouped-query attention models,
however, appear to be stable.


L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle : Open Foundation and Fine-Tuned Chat Models
Hugo TouvronLouis MartinyKevin Stoney
Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra
Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen
Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller
Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou
Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev
Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich
Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra
Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi
Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang

Llama 2: Open Foundation and Fine-Tuned Chat Models

Summary:
In this work, we develop and release Llama 2, a collection of pretrained and
fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70
billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for
dialogue use cases. Our models outperform open-source chat models on most
benchmarks we tested, and based on our human evaluations for helpfulness and
safety, may be a suitable substitute for closed-source models. We provide a
detailed description of our approach to fine-tuning and safety improvements of
Llama 2-Chat in order to enable the community to build on our work and
contribute to the responsible development of LLMs.

Hugo Touvron
Louis Martin
Kevin Stone
Peter Albert
Amjad Almahairi
Yasmine Babaei
Nikolay Bashlykov
Soumya Batra
Prajjwal Bhargava
Shruti Bhosale
Dan Bikel
Lukas Blecher
Cristian Canton Ferrer
Moya Chen
Guillem Cucurull
David Esiobu
Jude Fernandes
Jeremy Fu
Wenyin Fu
Brian Fuller
Cynthia Gao
Vedanuj Goswami
Naman Goyal
Anthony Hartshorn
Saghar Hosseini
Rui Hou
Hakan Inan
Marcin Kardas
Viktor Kerkez
Madian Khabsa
Isabel Kloumann
Artem Korenev
Punit Singh Koura
Marie-Anne Lachaux
Thibaut Lavril
Jenya Lee
Diana Liskovich
Yinghai Lu
Yuning Mao
Xavier Martinet
Todor Mihaylov
Pushkar Mishra
Igor Molybog
Yixin Nie
Andrew Poulton
Jeremy Reizenstein
Rashi Rungta
Kalyan Saladi
Alan Schelten
Ruan Silva
Eric Michael Smith
Ranjan Subramanian
Xiaoqing Ellen Tan
Binh Tang
Ross Taylor
Adina Williams
Jian Xiang Kuan
Puxin Xu
Zheng Yan
Iliyan Zarov
Yuchen Zhang
Angela Fan
Melanie Kambadur
Sharan Narang
Aurelien Rodriguez
Robert Stojnic
Sergey Edunov
Thomas Scialom


Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang
Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang
Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic
Sergey Edunov Thomas Scialom
GenAI, Meta
Abstract
In this work, we develop and release Llama 2, a collection of pretrained and ﬁne-tuned
large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
Our ﬁne-tuned LLMs, called L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , are optimized for dialogue use cases. Our
models outperform open-source chat models on most benchmarks we tested, and based on
ourhumanevaluationsforhelpfulnessandsafety,maybeasuitablesubstituteforclosedsource models. We provide a detailed description of our approach to ﬁne-tuning and safety


improvements of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc in order to enable the community to build on our work and
contribute to the responsible development of LLMs.
Equal contribution, corresponding authors: {tscialom, htouvron}@meta.com
ySecond author
Contributions for all the authors can be found in Section A.1.arXiv:2307.09288v2  [cs.CL]  19 Jul 2023
Contents
1 Introduction 3
2 Pretraining 5
2.1 Pretraining Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Training Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5


A.5 Data Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
A.6 Dataset Contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
A.7 Model Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
2
Figure1: Helpfulnesshumanevaluation resultsfor L/l.sc/a.sc/m.sc/a.sc
/two.taboldstyle-C/h.sc/a.sc/t.sccomparedtootheropen-sourceandclosed-source
models. Human raters compared model generations on ~4k
promptsconsistingofbothsingleandmulti-turnprompts.
The95%conﬁdenceintervalsforthisevaluationarebetween
1%and2%. MoredetailsinSection3.4.2. Whilereviewing


The95%conﬁdenceintervalsforthisevaluationarebetween
1%and2%. MoredetailsinSection3.4.2. Whilereviewing
these results, it is important to note that human evaluations
canbenoisyduetolimitationsofthepromptset,subjectivity
of the review guidelines, subjectivity of individual raters,
and the inherent diﬃculty of comparing generations.
Figure 2: Win-rate % for helpfulness and
safety between commercial-licensed baselines and L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , according to GPT4. Tocomplementthehumanevaluation,we
used a more capable model, not subject to
ourownguidance. Greenareaindicatesour
modelisbetteraccordingtoGPT-4. Toremove
ties, we used win= (win+loss). The orders in
whichthemodelresponsesarepresentedto
GPT-4arerandomlyswappedtoalleviatebias.
1 Introduction
Large Language Models (LLMs) have shown great promise as highly capable AI assistants that excel in
complex reasoning tasks requiring expert knowledge across a wide range of ﬁelds, including in specialized


complex reasoning tasks requiring expert knowledge across a wide range of ﬁelds, including in specialized
domains such as programming and creative writing. They enable interaction with humans through intuitive
chat interfaces, which has led to rapid and widespread adoption among the general public.
ThecapabilitiesofLLMsareremarkableconsideringtheseeminglystraightforwardnatureofthetraining
methodology. Auto-regressivetransformersarepretrainedonanextensivecorpusofself-superviseddata,
followed by alignment with human preferences via techniques such as Reinforcement Learning with Human
Feedback(RLHF).Althoughthetrainingmethodologyissimple,highcomputationalrequirementshave
limited the development of LLMs to a few players. There have been public releases of pretrained LLMs
(such as BLOOM (Scao et al., 2022), LLaMa-1 (Touvron et al., 2023), and Falcon (Penedo et al., 2023)) that
match the performance of closed pretrained competitors like GPT-3 (Brown et al., 2020) and Chinchilla
(Hoﬀmann et al., 2022), but none of these models are suitable substitutes for closed “product” LLMs, such
asChatGPT,BARD,andClaude. TheseclosedproductLLMsareheavilyﬁne-tunedtoalignwithhuman


asChatGPT,BARD,andClaude. TheseclosedproductLLMsareheavilyﬁne-tunedtoalignwithhuman
preferences, which greatly enhances their usability and safety. This step can require signiﬁcant costs in
computeandhumanannotation,andisoftennottransparentoreasilyreproducible,limitingprogresswithin
the community to advance AI alignment research.
In this work, we develop and release Llama 2, a family of pretrained and ﬁne-tuned LLMs, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle and
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested,
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc models generally perform better than existing open-source models. They also appear to
be on par with some of the closed-source models, at least on the human evaluations we performed (see


be on par with some of the closed-source models, at least on the human evaluations we performed (see
Figures1and3). Wehavetakenmeasurestoincreasethesafetyofthesemodels,usingsafety-speciﬁcdata
annotation and tuning, as well as conducting red-teaming and employing iterative evaluations. Additionally,
thispapercontributesathoroughdescriptionofourﬁne-tuningmethodologyandapproachtoimproving
LLM safety. We hope that this openness will enable the community to reproduce ﬁne-tuned LLMs and
continue to improve the safety of those models, paving the way for more responsible development of LLMs.
Wealsosharenovelobservationswemadeduringthedevelopmentof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc ,suchas
the emergence of tool usage and temporal organization of knowledge.
3


the emergence of tool usage and temporal organization of knowledge.
3
Figure 3: Safety human evaluation results for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc compared to other open-source and closedsource models. Human raters judged model generations for safety violations across ~2,000 adversarial
prompts consisting of both single and multi-turn prompts. More details can be found in Section 4.4. It is
importanttocaveatthesesafetyresultswiththeinherentbiasofLLMevaluationsduetolimitationsofthe
promptset,subjectivityofthereviewguidelines,andsubjectivityofindividualraters. Additionally,these
safety evaluations are performed using content standards that are likely to be biased towards the L/l.sc/a.sc/m.sc/a.sc
/two.taboldstyle-C/h.sc/a.sc/t.scmodels.
We are releasing the following models to the general public for research and commercial use‡:


We are releasing the following models to the general public for research and commercial use‡:
1.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle,anupdatedversionof L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle,trainedonanewmixofpubliclyavailabledata. Wealso
increasedthesizeofthepretrainingcorpusby40%,doubledthecontextlengthofthemodel,and
adoptedgrouped-queryattention(Ainslieetal.,2023). Wearereleasingvariantsof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle with
7B,13B,and70Bparameters. Wehavealsotrained34Bvariants,whichwereportoninthispaper
but are not releasing.§
2.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , a ﬁne-tuned version of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle that is optimized for dialogue use cases. We release


variants of this model with 7B, 13B, and 70B parameters as well.
WebelievethattheopenreleaseofLLMs,whendonesafely,willbeanetbeneﬁttosociety. LikeallLLMs,
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle is a new technology that carries potential risks with use (Bender et al., 2021b; Weidinger et al., 2021;
Solaimanet al.,2023). Testingconductedtodate hasbeeninEnglish andhasnot— andcouldnot— cover
all scenarios. Therefore, before deploying any applications of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , developers should perform
safetytestingand tuningtailoredtotheirspeciﬁcapplicationsofthemodel. Weprovidearesponsibleuse
guide¶and code examples‖to facilitate the safe deployment of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . More details of


our responsible release strategy can be found in Section 5.3.
Theremainderofthispaperdescribesourpretrainingmethodology(Section2),ﬁne-tuningmethodology
(Section 3), approach to model safety (Section 4), key observations and insights (Section 5), relevant related
work (Section 6), and conclusions (Section 7).
‡https://ai.meta.com/resources/models-and-libraries/llama/
§We are delaying the release of the 34B model due to a lack of time to suﬃciently red team.
¶https://ai.meta.com/llama
‖https://github.com/facebookresearch/llama
4
Figure 4: Training of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc : This process begins with the pretraining ofL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle using publicly
availableonlinesources. Followingthis,wecreateaninitialversionof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc throughtheapplication


ofsupervised ﬁne-tuning . Subsequently, the model is iteratively reﬁned using Reinforcement Learning
with Human Feedback (RLHF) methodologies, speciﬁcally through rejection sampling and Proximal Policy
Optimization(PPO).ThroughouttheRLHFstage,theaccumulationof iterativerewardmodelingdata in
parallel with model enhancements is crucial to ensure the reward models remain within distribution.
2 Pretraining
Tocreatethenewfamilyof L/l.sc/a.sc/m.sc/a.sc /two.taboldstylemodels,webeganwiththepretrainingapproachdescribedinTouvronetal.
(2023), using an optimized auto-regressive transformer, but made several changes to improve performance.
Speciﬁcally,weperformedmorerobustdatacleaning,updatedourdatamixes,trainedon40%moretotal
tokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability


tokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability
for our larger models. Table 1 compares the attributes of the new L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models with the L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle models.
2.1 Pretraining Data
Our training corpus includes a new mix of data from publicly available sources, which does not include data
fromMeta’sproductsorservices. Wemadeaneﬀorttoremovedatafromcertainsitesknowntocontaina
highvolumeofpersonalinformationaboutprivateindividuals. Wetrainedon2trilliontokensofdataasthis
providesagoodperformance–costtrade-oﬀ,up-samplingthemostfactualsourcesinaneﬀorttoincrease
knowledge and dampen hallucinations.
Weperformedavarietyofpretrainingdatainvestigationssothatuserscanbetterunderstandthepotential
capabilities and limitations of our models; results can be found in Section 4.1.
2.2 Training Details


capabilities and limitations of our models; results can be found in Section 4.1.
2.2 Training Details
We adopt most of the pretraining setting and model architecture from L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle . We use the standard
transformer architecture (Vaswani et al., 2017), apply pre-normalization using RMSNorm (Zhang and
Sennrich, 2019), use the SwiGLU activation function (Shazeer, 2020), and rotary positional embeddings
(RoPE, Su et al. 2022). The primary architectural diﬀerences from L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle include increased context length
andgrouped-queryattention(GQA).WedetailinAppendixSectionA.2.1eachofthesediﬀerenceswith
ablation experiments to demonstrate their importance.
Hyperparameters. We trained using the AdamW optimizer (Loshchilov and Hutter, 2017), with 1=
0:9;2= 0:95;eps= 105. We use a cosine learning rate schedule, with warmup of 2000 steps, and decay


0:9;2= 0:95;eps= 105. We use a cosine learning rate schedule, with warmup of 2000 steps, and decay
ﬁnallearningratedownto10%ofthepeaklearningrate. Weuseaweightdecayof 0:1andgradientclipping
of1:0. Figure 5 (a) shows the training loss for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle with these hyperparameters.
5
Training Data Params Context
LengthGQA Tokens LR
L/l.sc/a.sc/m.sc/a.sc /one.taboldstyleSee Touvron et al.
(2023)7B 2k 7 1.0T 3:0104
13B 2k 7 1.0T 3:0104
33B 2k 7 1.4T 1:5104
65B 2k 7 1.4T 1:5104
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyleA new mix of publicly
available online data7B 4k 7 2.0T 3:0104
13B 4k 7 2.0T 3:0104


available online data7B 4k 7 2.0T 3:0104
13B 4k 7 2.0T 3:0104
34B 4k X 2.0T 1:5104
70B 4k X 2.0T 1:5104
Table1: L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle familyofmodels. Tokencountsrefertopretrainingdataonly. Allmodelsaretrainedwith
a global batch-size of 4M tokens. Bigger models — 34B and 70B — use Grouped-Query Attention (GQA) for
improved inference scalability.
0 250 500 750 1000 1250 1500 1750 2000
Processed Tokens (Billions)1.41.51.61.71.81.92.02.12.2Train PPLLlama-2
7B
13B
34B
70B
Figure5: TrainingLossfor L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models. Wecomparethetraininglossof the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle family ofmodels.


We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation.
Tokenizer. Weusethesametokenizeras L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle;itemploysabytepairencoding(BPE)algorithm(Sennrich
etal.,2016)usingtheimplementationfromSentencePiece(KudoandRichardson,2018). Aswith L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle,
we split all numbers into individual digits and use bytes to decompose unknown UTF-8 characters. The total
vocabulary size is 32k tokens.
2.2.1 Training Hardware & Carbon Footprint
TrainingHardware. WepretrainedourmodelsonMeta’sResearchSuperCluster(RSC)(LeeandSengupta,
2022)aswellasinternalproductionclusters. BothclustersuseNVIDIAA100s. Therearetwokeydiﬀerences
between the two clusters, with the ﬁrst being the type of interconnect available: RSC uses NVIDIA Quantum
InﬁniBandwhileourproductionclusterisequippedwithaRoCE(RDMAoverconvergedEthernet)solution


InﬁniBandwhileourproductionclusterisequippedwithaRoCE(RDMAoverconvergedEthernet)solution
based on commodity ethernet Switches. Both of these solutions interconnect 200 Gbps end-points. The
seconddiﬀerenceistheper-GPUpowerconsumptioncap—RSCuses400Wwhileourproductioncluster
uses350W.Withthistwo-clustersetup,wewereabletocomparethesuitabilityofthesediﬀerenttypesof
interconnectforlargescaletraining. RoCE(whichisamoreaﬀordable,commercialinterconnectnetwork)
6
Time
(GPU hours)Power
Consumption (W)Carbon Emitted
(tCO 2eq)
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle7B 184320 400 31.22
13B 368640 400 62.44
34B 1038336 350 153.90
70B 1720320 400 291.42
Total 3311616 539.00
Table 2: CO 2emissions during pretraining. Time: total GPU time required for training each model. Power
Consumption: peak power capacity per GPU device for the GPUs used adjusted for power usage eﬃciency.


Consumption: peak power capacity per GPU device for the GPUs used adjusted for power usage eﬃciency.
100%oftheemissionsaredirectlyoﬀsetbyMeta’ssustainabilityprogram,andbecauseweareopenlyreleasing
these models, the pretraining costs do not need to be incurred by others.
can scale almost as well as expensive Inﬁniband up to 2000 GPUs, which makes pretraining even more
democratizable.
CarbonFootprintofPretraining. Followingprecedingresearch(Benderetal.,2021a;Pattersonetal.,2021;
Wu et al., 2022; Dodge et al., 2022) and using power consumption estimates of GPU devices and carbon
eﬃciency, we aim tocalculate thecarbon emissions resultingfrom the pretrainingof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models. The
actualpowerusageofaGPUisdependentonitsutilizationandislikelytovaryfromtheThermalDesign
Power(TDP)thatweemployasanestimationforGPUpower. Itisimportanttonotethatourcalculations
do not account for further power demands, such as those from interconnect or non-GPU server power


do not account for further power demands, such as those from interconnect or non-GPU server power
consumption,norfromdatacentercoolingsystems. Additionally,thecarbonoutputrelatedtotheproduction
of AI hardware, like GPUs, could add to the overall carbon footprint as suggested by Gupta et al. (2022b,a).
Table 2 summarizes the carbon emission for pretraining the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle family of models. A cumulative of
3.3M GPUhours ofcomputation wasperformed onhardware oftype A100-80GB (TDPof 400Wor 350W).
Weestimatethetotalemissionsfortrainingtobe 539tCO 2eq,ofwhich100%weredirectlyoﬀsetbyMeta’s
sustainability program.∗∗Our open release strategy also means that these pretraining costs will not need to
be incurred by other companies, saving more global resources.
2.3 L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle Pretrained Model Evaluation


2.3 L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle Pretrained Model Evaluation
In this section, we report the results for the L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle base models, MosaicML Pretrained
Transformer(MPT)††models,andFalcon(Almazroueietal.,2023)modelsonstandardacademicbenchmarks.
For all the evaluations, we use our internal evaluations library. We reproduce results for the MPT and Falcon
modelsinternally. Forthesemodels,wealwayspickthebestscorebetweenourevaluationframeworkand
any publicly reported results.
InTable3,wesummarizetheoverallperformanceacrossasuiteofpopularbenchmarks. Notethatsafety
benchmarks are shared in Section 4.1. The benchmarks are grouped into the categories listed below. The
results for all the individual benchmarks are available in Section A.2.2.
•Code.Wereporttheaveragepass@1scoresofourmodelsonHumanEval(Chenetal.,2021)and
MBPP (Austin et al., 2021).


•Code.Wereporttheaveragepass@1scoresofourmodelsonHumanEval(Chenetal.,2021)and
MBPP (Austin et al., 2021).
•CommonsenseReasoning. WereporttheaverageofPIQA(Bisketal.,2020),SIQA(Sapetal.,2019),
HellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge
(Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al.,
2018). We report 7-shot results for CommonSenseQA and 0-shot results for all other benchmarks.
•WorldKnowledge. Weevaluatethe5-shotperformanceonNaturalQuestions(Kwiatkowskietal.,
2019) and TriviaQA (Joshi et al., 2017) and report the average.
•Reading Comprehension. For reading comprehension, we report the 0-shot average on SQuAD
(Rajpurkar et al., 2018), QuAC (Choi et al., 2018), and BoolQ (Clark et al., 2019).
•MATH. We report the average of the GSM8K (8 shot) (Cobbe et al., 2021) and MATH (4 shot)


•MATH. We report the average of the GSM8K (8 shot) (Cobbe et al., 2021) and MATH (4 shot)
(Hendrycks et al., 2021) benchmarks at top 1.
∗∗https://sustainability.fb.com/2021-sustainability-report/
††https://www.mosaicml.com/blog/mpt-7b
7
Model Size CodeCommonsense
ReasoningWorld
KnowledgeReading
ComprehensionMath MMLU BBH AGI Eval
MPT7B 20.5 57.4 41.0 57.5 4.9 26.8 31.0 23.5
30B 28.9 64.9 50.0 64.7 9.1 46.9 38.0 33.8
Falcon7B 5.6 56.1 42.8 36.0 4.6 26.2 28.0 21.2
40B 15.2 69.2 56.7 65.7 12.6 55.4 37.1 37.0
L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle7B 14.1 60.8 46.2 58.5 6.95 35.1 30.3 23.9


•Popular Aggregated Benchmarks . We report the overall results for MMLU (5 shot) (Hendrycks
et al., 2020), Big Bench Hard (BBH) (3 shot) (Suzgun et al., 2022), and AGI Eval (3–5 shot) (Zhong
et al., 2023). For AGI Eval, we only evaluate on the English tasks and report the average.
As shown in Table 3, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models outperform L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle models. In particular, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B improves the
resultsonMMLUandBBHby 5and8points,respectively,comparedto L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle 65B.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 7Band30B
modelsoutperformMPTmodelsofthecorrespondingsizeonallcategoriesbesidescodebenchmarks. Forthe


modelsoutperformMPTmodelsofthecorrespondingsizeonallcategoriesbesidescodebenchmarks. Forthe
Falcon models, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 7B and 34B outperform Falcon 7B and 40B models on all categories of benchmarks.
Additionally, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B model outperforms all open-source models.
In addition to open-source models, we also compare L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B results to closed-source models. As shown
in Table 4, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a signiﬁcant
gaponcodingbenchmarks. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70BresultsareonparorbetterthanPaLM(540B)(Chowdheryetal.,


2022)onalmostallbenchmarks. Thereisstillalargegapinperformancebetween L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70BandGPT-4
and PaLM-2-L.
We also analysed the potential data contamination and share the details in Section A.6.
Benchmark (shots) GPT-3.5 GPT-4 PaLM PaLM-2-L L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle
MMLU (5-shot) 70.0 86.4 69.3 78.3 68.9
TriviaQA (1-shot) – – 81.4 86.1 85.0
Natural Questions (1-shot) – – 29.3 37.5 33.0
GSM8K (8-shot) 57.1 92.0 56.5 80.7 56.8
HumanEval (0-shot) 48.1 67.0 26.2 – 29.9
BIG-Bench Hard (3-shot) – – 52.3 65.7 51.2
Table 4: Comparison to closed-source models on academic benchmarks. Results for GPT-3.5 and GPT-4


Table 4: Comparison to closed-source models on academic benchmarks. Results for GPT-3.5 and GPT-4
are from OpenAI (2023). Results for the PaLM model are from Chowdhery et al. (2022). Results for the
PaLM-2-L are from Anil et al. (2023).
3 Fine-tuning
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is the result of several months of research and iterative applications of alignment techniques,
including both instruction tuning and RLHF, requiring signiﬁcant computational and annotation resources.
In this section, we report on our experiments and ﬁndings using supervised ﬁne-tuning (Section 3.1), as
well as initial and iterative reward modeling (Section 3.2.2) and RLHF (Section 3.2.3). We also share a
new technique, Ghost Attention (GAtt), which we ﬁnd helps control dialogue ﬂow over multiple turns
(Section 3.3). See Section 4.2 for safety evaluations on ﬁne-tuned models.
8
3.1 Supervised Fine-Tuning (SFT)


(Section 3.3). See Section 4.2 for safety evaluations on ﬁne-tuned models.
8
3.1 Supervised Fine-Tuning (SFT)
äPrompt: Writeapoemtohelpmeremembertheﬁrst10elementsontheperiodictable,givingeach
element its own line.
Response: Hydrogen comes ﬁrst as element number one.
Helium is second for balloons to have fun!
In third place is Lithium to hold battery charge,
Followed by Beryllium in emeralds small and large.
Boron’s number ﬁve to help us keep things clean.
Carbon’s next at six, and it’s found in every gene.
Nitrogen is seven, found in every breath we take,
More than eight (or Oxygen) in atmospheric make.
Number nine is Fluorine, helping polish up our teeth.
Neon gives us glowing signs to read and stand beneath.
äPrompt: I want you to roast me. I want you to make it particularly brutal, swearing at me.
Response: I’m sorry, but I cannot comply with that request. Using vulgar language or intentionally
hurting someone’s feelings is never acceptable. Instead of asking for a roast, I suggest


hurting someone’s feelings is never acceptable. Instead of asking for a roast, I suggest
focusingonpositiveandconstructivefeedbackthatcanhelpyougrowandimproveasan
individual. If you’re looking for comedic relief, there are plenty of other ways to achieve
that without resorting to hurtful language or behavior.
Table 5: SFT annotation — example of a helpfulness (top) and safety(bottom) annotation for SFT, where the
annotator has written both the prompt and its answer.
Getting Started. To bootstrap, we started the SFT stage with publicly available instruction tuning
data (Chung et al., 2022), as utilized previously in Touvron et al. (2023).
QualityIsAllYouNeed. Third-partySFTdataisavailablefrommanydiﬀerentsources,butwefoundthat
manyofthesehaveinsuﬃcientdiversityandquality—inparticularforaligningLLMstowardsdialogue-style
instructions. As a result, we focused ﬁrst on collecting several thousand examples of high-quality SFT data,
as illustrated in Table 5. By setting aside millions of examples from third-party datasets and using fewer but
higher-quality examples from our own vendor-based annotation eﬀorts, our results notably improved. These


higher-quality examples from our own vendor-based annotation eﬀorts, our results notably improved. These
ﬁndingsaresimilarinspirittoZhouetal.(2023),whichalsoﬁndsthatalimitedsetofcleaninstruction-tuning
datacanbesuﬃcienttoreachahighlevelofquality. WefoundthatSFTannotationsintheorderoftensof
thousands was enough to achieve a high-quality result. We stopped annotating SFT after collecting a total of
27,540 annotations. Note that we do not include any Meta user data.
Wealsoobservedthatdiﬀerentannotationplatformsandvendorscanresultinmarkedlydiﬀerentdownstream model performance, highlighting the importance of data checks even when using vendors to source
annotations. Tovalidateourdataquality,wecarefullyexaminedasetof180examples,comparingtheannotations provided by humans with the samples generated by the model through manual scrutiny. Surprisingly,
we found that the outputs sampled from the resulting SFT model were often competitive with SFT data
handwritten by human annotators, suggesting that we could reprioritize and devote more annotation eﬀort
to preference-based annotation for RLHF.


handwritten by human annotators, suggesting that we could reprioritize and devote more annotation eﬀort
to preference-based annotation for RLHF.
Fine-Tuning Details. For supervised ﬁne-tuning, we use a cosine learning rate schedule with an initial
learning rate of 2105, a weight decay of 0.1, a batch size of 64, and a sequence length of 4096 tokens.
For the ﬁne-tuning process, each sample consists of a prompt and an answer. To ensure the model sequence
lengthisproperlyﬁlled,weconcatenateallthepromptsandanswersfromthetrainingset. Aspecialtokenis
utilizedtoseparatethepromptandanswersegments. Weutilizeanautoregressiveobjectiveandzero-out
the loss on tokens from the user prompt, so as a result, we backpropagate only on answer tokens. Finally, we
ﬁne-tune the model for 2 epochs.
3.2 Reinforcement Learning with Human Feedback (RLHF)
RLHFisamodeltrainingprocedurethatisappliedtoaﬁne-tunedlanguagemodeltofurther alignmodel
behavior with human preferences and instruction following. We collect data that represents empirically
9


behavior with human preferences and instruction following. We collect data that represents empirically
9
sampled human preferences, whereby human annotators select which of two model outputs they prefer.
This human feedback is subsequently used to train a reward model, which learns patterns in the preferences
of the human annotators and can then automate preference decisions.
3.2.1 Human Preference Data Collection
Next, wecollect human preference datafor reward modeling. We chose a binarycomparison protocol over
other schemes, mainly because it enables us to maximize the diversity of collected prompts. Still, other
strategies are worth considering, which we leave for future work.
Our annotation procedure proceeds as follows. We ask annotators to ﬁrst write a prompt, then choose
betweentwosampledmodelresponses,basedonprovidedcriteria. Inordertomaximizethediversity,the
tworesponsestoagivenpromptaresampledfromtwodiﬀerentmodelvariants,andvaryingthetemperature
hyper-parameter. Inadditiontogivingparticipantsaforcedchoice,wealsoaskannotatorstolabelthedegree
to which they prefer their chosen response over the alternative: either their choice is signiﬁcantly better ,better,
slightly better , ornegligibly better/ unsure .


to which they prefer their chosen response over the alternative: either their choice is signiﬁcantly better ,better,
slightly better , ornegligibly better/ unsure .
For our collection of preference annotations, we focus on helpfulness and safety. Helpfulness refers to how
well L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc responses fulﬁll users’ requests and provide requested information; safety refers to
whether L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc ’s responses are unsafe, e.g., “giving detailed instructions on making a bomb” could
be considered helpful but is unsafe according to our safety guidelines. Separating the two allows us to
applyspeciﬁcguidelinestoeachandbetterguideannotators;forexample,oursafetyannotationsprovide
instructions to focus on adversarial prompts, among other guidance.
Apart from diﬀerences in annotation guidelines, we additionally collect a safety label during the safety stage.
This additional information bins model responses into one of three categories: 1) the preferred response


Apart from diﬀerences in annotation guidelines, we additionally collect a safety label during the safety stage.
This additional information bins model responses into one of three categories: 1) the preferred response
is safe and the other response is not, 2) both responses are safe, and 3) both responses are unsafe, with
18%, 47%, and 35% of the safety dataset falling into each bin, respectively. We do not include any examples
where the chosen response was unsafe and the other response safe, as we believe safer responses will also be
better/preferred by humans. Safety guidelines and more detailed information regarding safety annotations
can be found in Section 4.2.1.
Human annotations were collected in batches on a weekly basis. As we collected more preference data, our
reward models improved, and we were able to train progressively better versions for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc (see
theresultsinSection5,Figure20). L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc improvementalsoshiftedthemodel’sdatadistribution.
Since reward model accuracy can quickly degrade if not exposed to this new sample distribution, i.e., from


Since reward model accuracy can quickly degrade if not exposed to this new sample distribution, i.e., from
hyper-specialization(Scialometal.,2020b),itisimportantbeforeanew L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc tuningiterationto
gather new preference data using the latest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc iterations. This step helps keep the reward model
on-distribution and maintain an accurate reward for the latest model.
InTable6,wereportthestatisticsofrewardmodelingdatathatwecollectedovertime,andpresentthem
against multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al.,
2022a), OpenAISummarize(Stiennon etal., 2020),OpenAI WebGPT(Nakanoet al.,2021), StackExchange
(Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).


(Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).
We collected a large dataset ofover 1million binary comparisons based on humansapplyingour speciﬁed
guidelines, which we refer to as Metareward modeling data. Note that the number of tokens in prompts and
answers diﬀers depending on the text domain. Summarization and online forum data generally have longer
prompts, while dialogue-style prompts are usually shorter. Compared to existing open-source datasets, our
preference data features more conversation turns, and are longer, on average.
3.2.2 Reward Modeling
The reward model takes a model response and its corresponding prompt (including contexts from previous
turns) as inputs and outputs a scalar score to indicate the quality (e.g., helpfulness and safety) of the model
generation. Leveragingsuchresponsescoresasrewards,wecanoptimize L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc duringRLHFfor
better human preference alignment and improved helpfulness and safety.


better human preference alignment and improved helpfulness and safety.
Others have found that helpfulness and safety sometimes trade oﬀ (Bai et al., 2022a), which can make it
challengingforasinglerewardmodeltoperformwellonboth. Toaddressthis,wetraintwoseparatereward
models, one optimized for helpfulness (referred to as Helpfulness RM ) and another for safety ( Safety RM ).
We initialize our reward models from pretrained chat model checkpoints, as it ensures that both models
beneﬁtfromknowledgeacquiredinpretraining. Inshort, therewardmodel“knows” whatthechatmodel
10
DatasetNum. of
ComparisonsAvg. # Turns
per DialogueAvg. # Tokens
per ExampleAvg. # Tokens
in PromptAvg. # Tokens
in Response
Anthropic Helpful 122,387 3.0 251.5 17.7 88.4
Anthropic Harmless 43,966 3.0 152.5 15.7 46.4
OpenAI Summarize 176,625 1.0 371.1 336.0 35.1
OpenAI WebGPT 13,333 1.0 237.2 48.3 188.9
StackExchange 1,038,480 1.0 440.2 200.1 240.2


StackExchange 1,038,480 1.0 440.2 200.1 240.2
Stanford SHP 74,882 1.0 338.3 199.5 138.8
Synthetic GPT-J 33,139 1.0 123.3 13.0 110.3
Meta (Safety & Helpfulness) 1,418,091 3.9 798.5 31.4 234.1
Total 2,919,326 1.6 595.7 108.2 216.9
Table 6: Statistics of human preference data for reward modeling. We list both the open-source and
internally collected human preference data used for reward modeling. Note that a binary human preference
comparisoncontains2responses(chosenandrejected)sharingthesameprompt(andpreviousdialogue).
Eachexampleconsistsofaprompt(includingpreviousdialogueifavailable)andaresponse,whichisthe
inputoftherewardmodel. Wereportthenumberofcomparisons,theaveragenumberofturnsperdialogue,
the average number of tokens per example, per prompt and per response. More details on Meta helpfulness
and safety data per batch can be found in Appendix A.3.1.
knows. Thispreventscaseswhere, forinstance, thetwomodelswouldhaveaninformationmismatch, which


knows. Thispreventscaseswhere, forinstance, thetwomodelswouldhaveaninformationmismatch, which
could result in favoring hallucinations. The model architecture and hyper-parameters are identical to those
ofthepretrainedlanguagemodels,exceptthattheclassiﬁcationheadfornext-tokenpredictionisreplaced
with a regression head for outputting a scalar reward.
Training Objectives. To train the reward model, we convert our collected pairwise human preference data
into a binary ranking label format (i.e., chosen & rejected) and enforce the chosen response to have a higher
score than its counterpart. We used a binary ranking loss consistent with Ouyang et al. (2022):
Lranking =log((r(x;yc)r(x;yr))) (1)
wherer(x;y)is the scalar score output for prompt xand completion ywith model weights .ycis the
preferred response that annotators choose and yris the rejected counterpart.
Built on top of this binary ranking loss, we further modify it separately for better helpfulness and safety
rewardmodelsasfollows. Giventhatourpreferenceratingsisdecomposedasascaleoffourpoints(e.g.,


rewardmodelsasfollows. Giventhatourpreferenceratingsisdecomposedasascaleoffourpoints(e.g.,
signiﬁcantly better ), as presented in Section 3.2.1, it can be useful to leverage this information to explicitly
teachtherewardmodeltoassignmorediscrepantscorestothegenerationsthathavemorediﬀerences. To
do so, we further add a margin component in the loss:
Lranking =log((r(x;yc)r(x;yr)m(r))) (2)
where the margin m(r)is a discrete function of the preference rating. Naturally, we use a large margin
for pairs with distinct responses, and a smaller one for those with similar responses (shown in Table 27).
WefoundthismargincomponentcanimproveHelpfulnessrewardmodelaccuracyespeciallyonsamples
wheretworesponsesaremoreseparable. MoredetailedablationandanalysiscanbefoundinTable28in
Appendix A.3.3.
DataComposition. Wecombineournewlycollecteddatawithexistingopen-sourcepreferencedatasets


Appendix A.3.3.
DataComposition. Wecombineournewlycollecteddatawithexistingopen-sourcepreferencedatasets
toformalargertrainingdataset. Initially,open-sourcedatasetswereusedtobootstrapourrewardmodels
whilewewereintheprocessofcollectingpreferenceannotationdata. WenotethatinthecontextofRLHFin
this study, the role of reward signals is to learn human preference for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc outputs rather than
any model outputs. However, in our experiments, we do not observe negative transfer from the open-source
preferencedatasets. Thus,wehavedecidedtokeeptheminourdatamixture,astheycouldenablebetter
generalization for the reward model and prevent reward hacking, i.e. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc taking advantage of
some weaknesses of our reward, and so artiﬁcially inﬂating the score despite performing less well.


some weaknesses of our reward, and so artiﬁcially inﬂating the score despite performing less well.
With training data available from diﬀerent sources, we experimented with diﬀerent mixing recipes for both
HelpfulnessandSafetyrewardmodelstoascertainthebestsettings. Afterextensiveexperimentation,the
11
Helpfulness reward model is eventually trained on all Meta Helpfulness data, combined with an equal
partsoftheremainingdatauniformlysampledfromMetaSafetyandfromtheopen-sourcedatasets. The
Meta Safety reward model is trained on all Meta Safety and Anthropic Harmless data, mixed with Meta
Helpfulnessandopen-sourcehelpfulnessdataina90/10proportion. Wefoundthatthesettingwith10%
helpfulness data is especially beneﬁcial for the accuracy on samples where both the chosen and rejected
responses were deemed safe.
Training Details. We train for one epoch over the training data. In earlier experiments, we found that
traininglongercanleadtoover-ﬁtting. Weusethesameoptimizerparametersasforthebasemodel. The


traininglongercanleadtoover-ﬁtting. Weusethesameoptimizerparametersasforthebasemodel. The
maximum learning rate is 5106for the 70B parameter L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc and1105for the rest. The
learningrateisdecreasedonacosinelearningrateschedule,downto10%ofthemaximumlearningrate.
We use a warm-up of 3% of the total number of steps, with a minimum of 5. The eﬀective batch size is kept
ﬁxed at 512 pairs, or 1024 rows per batch.
Meta
Helpful.Meta
SafetyAnthropic
HelpfulAnthropic
HarmlessOpenAI
Summ.Stanford
SHPAvg
SteamSHP-XL 52.8 43.8 66.8 34.2 54.7 75.7 55.3
Open Assistant 53.8 53.4 67.7 68.4 71.7 55.0 63.0
GPT4 58.6 58.1 - - - - Safety RM 56.2 64.5 55.4 74.7 71.7 65.2 64.3


GPT4 58.6 58.1 - - - - Safety RM 56.2 64.5 55.4 74.7 71.7 65.2 64.3
Helpfulness RM 63.2 62.8 72.0 71.0 75.5 80.0 70.6
Table7: Rewardmodelresults. Performanceofourﬁnalhelpfulnessandsafetyrewardmodelsonadiverse
set of human preference benchmarks. Note that our model is ﬁne-tuned on our collected data, as opposed to
the other baselines that we report.
Test SetSigniﬁcantly
BetterBetterSlightly
BetterNegligibly
Better / UnsureAvg
Safety RMMeta Safety94.3 76.3 65.7 55.3 64.5
Helpfulness RM 89.9 73.2 63.8 54.5 62.8
Safety RMMeta Helpful.64.6 57.5 53.8 52.2 56.2
Helpfulness RM 80.7 67.5 60.9 54.7 63.2
Table 8: Granular reward model accuracy per preference rating. We report per-preference rating accuracy
forbothHelpfulnessandSafetyrewardmodelsontheMetaHelpfulnessandSafetytestsets. Thereward
models show superior accuracy on more distinct responses (e.g., signiﬁcantly better) and lower accuracy on


models show superior accuracy on more distinct responses (e.g., signiﬁcantly better) and lower accuracy on
similar responses (e.g., negligibly better).
Reward Model Results. On each batch of human preference annotation for reward modeling, we held out
1000examplesasatestsettoevaluateourmodels. Werefertotheunionofallpromptsforthecorresponding
test sets as “Meta Helpfulness” and “Meta Safety,” respectively.
As reference points, we also evaluated other publicly available alternatives as baselines: SteamSHP-XL
(Ethayarajh et al., 2022) based on FLAN-T5-xl, the Open Assistant (Köpf et al., 2023) reward model based on
DeBERTa V3 Large (He et al., 2020), and GPT4 accessible through the OpenAI’s API. Note that at inference
time, asopposedtotraining, alltherewardmodelscanpredictascalarforasingleoutput, withoutrequiring
to access its paired output. For GPT-4, we prompt with a zero-shot question “Choose the best answer between A
and B,”where A and B are the two responses for comparison.
We report the results in terms of accuracy in Table 7. As expected, our own reward models perform the best


and B,”where A and B are the two responses for comparison.
We report the results in terms of accuracy in Table 7. As expected, our own reward models perform the best
on our internaltest sets collected based on L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , with the Helpfulnessrewardmodel performing
bestontheMetaHelpfulnesstestset,andsimilarlytheSafetyrewardmodelperformingbestontheMeta
Safetytestset. Overall,ourrewardmodelsoutperformallofthebaselines,includingGPT-4. Interestingly,
GPT-4 performs better than other non-Meta reward models, despite not being trained directly nor targeting
speciﬁcally this reward modeling task.
12
1234567891011121314
Meta Helpfulness Data Batch Stage0.520.540.560.580.600.620.64Accuracy On All Examples
 7b
13b
70b
GPT4
OpenAssistant
1234567891011121314
Meta Helpfulness Data Batch Stage0.500.550.600.650.700.750.80Accuracy On Examples With Label "Significantly Better"
7b
13b
70b
GPT4


7b
13b
70b
GPT4
OpenAssistantFigure 6: Scaling trends for the reward model. More data and a larger-size model generally improve
accuracy, and it appears that our models have not yet saturated from learning on the training data.
Thefactthathelpfulnessandsafetyperformedthe bestontheirowndomainispotentiallyduetothetension
betweenthetwoobjectives(i.e.,beingashelpfulaspossibleversusrefusingunsafepromptswhennecessary),
whichmayconfusetherewardmodelduringtraining. Inorderforasinglemodeltoperformwellonboth
dimensions, it needs to not only learn to select the better response given a prompt but also to distinguish
adversarial prompts from safe ones. As a result, optimizing two separate models eases the reward modeling
task. More detailed analysis on this tension between safety and helpfulness can be found in Appendix A.4.1.
WhenwegroupthescoresbypreferenceratinginTable8,wecanseethattheaccuracyissuperiorforthe
“signiﬁcantlybetter”testsetanddegradesgraduallyascomparisonpairsbecomemoresimilar(e.g.,“slightly
better”). It is expected that learning to model human preferences becomes challenging when deciding


better”). It is expected that learning to model human preferences becomes challenging when deciding
betweentwosimilarmodelresponses,duetoannotatorsubjectivityandtheirrelianceonnuanceddetails
thatmaydiﬀerentiateresponses. Weemphasizethattheaccuracyonmoredistinctresponsesmattersthe
mosttoimprove L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc performance. Thehumanpreferenceannotationagreementrateisalsohigher
on more distinct responses than similar pairs.
Scaling Trends. Westudythescalingtrendsintermsofdataandmodelsizefortherewardmodel,ﬁnetuning diﬀerent model sizes on an increasing amount of the reward model data collected each week (see the
detailsonvolumeperbatchinTable26). Figure6reportsthesetrends,showingtheexpectedresultthatlarger
models obtain higher performance for a similar volume of data. More importantly, the scaling performance
hasnotyetplateauedgiventheexistingvolumeofdataannotationusedfortraining,asignalthatthereis
room for more improvement with more annotations. We note that reward model accuracy is one of the most


room for more improvement with more annotations. We note that reward model accuracy is one of the most
important proxies for the ﬁnal performance of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . While best practices for comprehensively
evaluating a generative model is an open research question, the ranking task of the reward has no ambiguity.
Therefore, everything else being equal, an improvement of the reward model can be directly translated into
an improvement for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .
3.2.3 Iterative Fine-Tuning
As we received more batches of human preference data annotation, we were able to train better reward
modelsandcollectmoreprompts. WethereforetrainedsuccessiveversionsforRLHFmodels,referredto
here as RLHF-V1, ..., RLHF-V5.
We explored RLHF ﬁne-tuning with two main algorithms:
•Proximal Policy Optimization (PPO) (Schulman et al., 2017), the standard in RLHF literature.


•Proximal Policy Optimization (PPO) (Schulman et al., 2017), the standard in RLHF literature.
•RejectionSamplingﬁne-tuning . Wesample Koutputsfromthemodelandselectthebestcandidate
with our reward, consistent with Bai et al. (2022b). The same re-ranking strategy for LLMs was also
proposedinDengetal.(2019),wheretherewardisseenasanenergyfunction. Here,wegoonestep
further,anduse theselectedoutputsfora gradientupdate. For eachprompt,thesample obtaining
13
100101
N Samples0.540.560.580.600.620.640.66Reward Score
Max of the rewards
Median of the rewardsFigure 7: Max and median reward among N samples ,N2[1;:::; 100]averaged over our training set of
prompts. The delta between max and median can be interpreted as potential gain with Rejection Sampling.
the highestrewardscore is consideredthe newgold standard. Similar toScialom etal. (2020a), we
then ﬁne-tune our model on the new set of ranked samples, reinforcing the reward.
The two RL algorithms mainly diﬀer in:


then ﬁne-tune our model on the new set of ranked samples, reinforcing the reward.
The two RL algorithms mainly diﬀer in:
•Breadth— in Rejection Sampling, the model explores Ksamples for a given prompt, while only one
generation is done for PPO.
•Depth— in PPO, during training at step tthe sample is a function of the updated model policy from
t1afterthegradientupdateofthepreviousstep. InRejectionSamplingﬁne-tuning,wesample
all the outputs given the initial policy of our model to collect a new dataset, before applying the
ﬁne-tuning similar to SFT. However, since we applied iterative model updates, the fundamental
diﬀerences between the two RL algorithms are less pronounced.
Until RLHF (V4), we used only Rejection Sampling ﬁne-tuning, and after that, we combined the two
sequentially, applying PPO on top of the resulted Rejection Sampling checkpoint before sampling again.
100101102
Number Samples0.10.20.30.40.50.6Reward Score
SFT
100101102
Number Samples0.350.400.450.500.550.600.650.70Reward Score
RLHF


SFT
100101102
Number Samples0.350.400.450.500.550.600.650.70Reward Score
RLHF
reward_max (T=0.6)
reward_max (T=0.8)
reward_max (T=0.9)
reward_max (T=1)
reward_max (T=1.1)
reward_max (T=1.2)
reward_max (T=1.3)
reward_max (T=1.4)
reward_max (T=1.5)
Figure 8: RLHF impact of the temperature when sampling N outputs and scoring them with a reward
model.
Rejection Sampling. We perform rejection sampling only with our largest 70B L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . All smaller
models are ﬁne-tuned on rejection sampled data from the larger model, thus distilling the large-model
capabilities into the smaller ones. We leave further analysis of the eﬀect of this distillation for future work.


capabilities into the smaller ones. We leave further analysis of the eﬀect of this distillation for future work.
Ateachiterativestage,wesample Kanswersforeachpromptfromthemostrecentmodel. Wescoreeach
sample given the best reward model accessible at the time of the experiment, and then select the best answer
foragivenprompt. Inearlierversionsofourmodel,uptoRLHFV3,ourapproachwastoconﬁneanswer
selection solely to the“bag” of samples gathered from the precedingiteration. For example, RLHF V3was
trained using only samples from RLHF V2. However, despite continuous improvement, this method led to a
14
regressionin somecapabilities. Forexample,RLHFV3 struggledmore thanpreviousversionstocompose
rhyming lines in poems, as discerned through qualitative analysis, suggesting that further investigation into
the causes of and mitigations for forgetting (Kirkpatrick et al., 2017; Nguyen et al., 2019; Ramasesh et al.,
2021) could be a fruitful area for additional future research.
In response, on subsequent iterations, we modiﬁed our strategy, incorporating top-performing samples from


2021) could be a fruitful area for additional future research.
In response, on subsequent iterations, we modiﬁed our strategy, incorporating top-performing samples from
all prior iterations, such as those used in RLHF-V1 and RLHF-V2. Although we do not present speciﬁc
ﬁgures, this adjustment demonstrated considerable enhancements in performance and eﬀectively addressed
the previously noted issues. This mitigation can be seen as analogous to Synnaeve et al. (2019) and Vinyals
et al. (2019) in the RL literature.
We illustrate the beneﬁt of Rejection Sampling in Figure 7. The delta between the maximum and median
curves can be interpreted as the potential gain of ﬁne-tuning on the best output. As expected, this delta
increases with more samples, since the maximum increases (i.e., more samples, more opportunities to
generateagoodtrajectory),whilethemedianremainsstationary. Thereisadirectconnectionbetweenthe
explorationand themaximum rewardwe canobtain amongthesamples. Thetemperatureparameteralso
plays an important role for exploration, as a higher temperature enables us to sample more diverse outputs.


plays an important role for exploration, as a higher temperature enables us to sample more diverse outputs.
In Figure 8, we report for a L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -SFT (left) and a L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -RLHF (right), the maximum
rewardcurvesamongNsamples(with N2[1;:::; 100]),fordiﬀerenttemperatures. Wecanobservethat
theoptimaltemperatureisnotconstantduringtheiterativemodelupdates: RLHFhasadirectimpacton
rescalingthetemperature. For L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -RLHF,theoptimaltemperaturewhensamplingbetween10
and 100 outputs is T2[1:2;1:3]. Given a ﬁnite compute budget, it is therefore necessary to re-adjust the


and 100 outputs is T2[1:2;1:3]. Given a ﬁnite compute budget, it is therefore necessary to re-adjust the
temperatureprogressively. Note thatthistemperature rescalinghappensfor aconstantnumber ofstepsfor
each model, and always starting from the base model on each new RLHF version.
PPO.WefurthertrainourlanguagemodelfollowingtheRLschemeofStiennonetal.(2020),whichusesthe
reward model as an estimate for the true reward function (human preference) and the pretrained language
model as the policy to optimize. During this phase, we seek to optimize the following objective:
arg max
EpD;g[R(gjp)] (3)
We iteratively improve the policy by sampling prompts pfrom our dataset Dand generations gfrom the
policyand use the PPO algorithm and loss function to achieve this objective.
The ﬁnal reward function we use during optimization,
R(gjp) =~Rc(gjp)DKL((gjp)k0(gjp)) (4)
contains a penalty term for diverging from the original policy 0. As was observed in other works (Stiennon


contains a penalty term for diverging from the original policy 0. As was observed in other works (Stiennon
et al., 2020; Ouyang et al., 2022), we ﬁnd this constraint is useful for training stability, and to reduce reward
hackingwherebywewouldachievehighscoresfromtherewardmodelbutlowscoresfromhumanevaluation.
We deﬁneRcto be a piecewise combination of the safety ( Rs) and helpfulness ( Rh) reward models. We
havetaggedpromptsinourdatasetthatmightelicitpotentiallyunsaferesponsesandprioritizethescores
from the safety model. The threshold of 0:15is chosen for ﬁltering unsafe responses, corresponding to a
precisionof 0:89andarecallof 0:55evaluatedontheMetaSafetytestset. Wealsoﬁnditimportanttowhiten
the ﬁnal linear scores (shown here by reversing the sigmoid with the logit function) in order to increase
stability and balance properly with the KL penalty term ( ) above.


stability and balance properly with the KL penalty term ( ) above.
Rc(gjp) =Rs(gjp)if/i.sc/s.sc_/s.sc/a.sc/f.sc/e.sc/t.sc/y.sc (p)orRs(gjp)<0:15
Rh(gjp)otherwise
~Rc(gjp) =/w.sc/h.sc/i.sc/t.sc/e.sc/n.sc (/l.sc/o.sc/g.sc/i.sc/t.sc (Rc(gjp)))
For all models, we use the AdamW optimizer (Loshchilov and Hutter, 2017), with 1= 0:9;2= 0:95;eps=
105. Weuseaweightdecayof 0:1,gradientclippingof 1:0,andaconstantlearningrateof 106. Foreach
PPOiterationweuseabatchsizeof 512,aPPOclipthresholdof 0:2,amini-batchsizeof 64,andtakeone
gradient step per mini-batch. For the 7B and 13B models, we set = 0:01(KL penalty), and for the 34B and


gradient step per mini-batch. For the 7B and 13B models, we set = 0:01(KL penalty), and for the 34B and
70B models, we set = 0:005.
15
Figure 9: Issues with multi-turn memory (left)can be improved with GAtt (right).
We train for between 200and400iterations for all our models, and use evaluations on held-out prompts for
earlystopping. EachiterationofPPOonthe70Bmodeltakesonaverage 330seconds. Totrainquicklywith
large batch sizes, we use FSDP (Zhao et al., 2023). This was eﬀective when using O(1) forward or backward
passes,butcausedalargeslowdown( 20)duringgeneration,evenwhenusingalargebatchsizeandKV
cache. We were able to mitigate this by consolidating the model weights to each node once before generation
and then freeing the memory after generation, resuming the rest of the training loop.
3.3 System Message for Multi-Turn Consistency
In a dialogue setup, some instructions should apply for all the conversation turns, e.g., to respond succinctly,


3.3 System Message for Multi-Turn Consistency
In a dialogue setup, some instructions should apply for all the conversation turns, e.g., to respond succinctly,
or to“act as”some public ﬁgure. When we provided such instructions to L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , the subsequent
response should always respect the constraint. However, our initial RLHF models tended to forget the initial
instruction after a few turns of dialogue, as illustrated in Figure 9 (left).
To address these limitations, we propose Ghost Attention (GAtt), a very simple method inspired by Context
Distillation (Bai et al., 2022b) that hacks the ﬁne-tuning data to help the attention focus in a multi-stage
process. GAtt enables dialogue control over multiple turns, as illustrated in Figure 9 (right).
GAttMethod. Assumewe haveaccess toa multi-turndialoguedataset betweentwo persons(e.g., auser
and an assistant), with a list of messages [u1;a1;:::;un;an], whereunandancorrespond to the user and


and an assistant), with a list of messages [u1;a1;:::;un;an], whereunandancorrespond to the user and
assistant messages for turn n, respectively. Then, we deﬁne an instruction, inst, that should be respected
throughout the dialogue. For example, instcould be “act as.” We can then synthetically concatenate this
instruction to all the user messages of the conversation.
Next, we can sample from this synthetic data using the latest RLHF model. We now have a context-dialogue
andthesamplewithwhichtoﬁne-tuneamodel,inaprocessanalogoustoRejectionSampling. Insteadof
augmentingallcontext-dialogueturnswiththeinstruction,wecandropitinallbuttheﬁrstturn,butthis
wouldleadtoamismatchattrainingtimebetweenthesystemmessage,i.e.,alltheintermediateassistant
messages that come before the last turn, and our sample. To ﬁx this issue, which could hurt the training, we
simply set the loss to 0 for all the tokens from the previous turns, including assistant messages.
For the training instructions, we created a few synthetic constraints to sample from: Hobbies ( “You enjoy


For the training instructions, we created a few synthetic constraints to sample from: Hobbies ( “You enjoy
e.g. Tennis” ),Language ( “Speakine.g. French” ),or PublicFigure( “Actase.g. Napoleon” ). Toobtainthelists
of hobbies and public ﬁgures, we asked L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc to generate it, avoiding a mismatch between the
instructionandmodelknowledge(e.g.,askingthemodeltoactassomeoneithadnotencounteredduring
training). Tomaketheinstructionsmorecomplexanddiverse,weconstructtheﬁnalinstructionbyrandomly
combining the above constraints. When constructing the ﬁnal system message for the training data, we also
16
modifytheoriginalinstructionhalfofthetimetobelessverbose,e.g., “AlwaysactasNapoleonfromnow” ->


16
modifytheoriginalinstructionhalfofthetimetobelessverbose,e.g., “AlwaysactasNapoleonfromnow” ->
”Figure: Napoleon.” These steps produce an SFT dataset, on which we can ﬁne-tune L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .
GAtt Evaluation. We applied GAtt after RLHF V3. We report a quantitative analysis indicating that GAtt
isconsistentupto20+turns,untilthemaximumcontextlengthisreached(seeAppendixA.3.5). Wetriedto
setconstraints notpresentinthetraining ofGAttatinference time,forinstance “Always answerwithHaiku,”
for which the model remained consistent as illustrated in Appendix Figure 28.
Act as Oscar
Wilde
Give short
answers Wh
ich is the
best city to
live in?
London, of
course!
Why is that
? My dear
fellow, London
is the ep
itome of
sophistic
ation and culture
. It is
the city of
Shakespeare and Dick
ens, of
the great univers
ities, of
the museums
and galleries
. It is
the city of


. It is
the city of
Shakespeare and Dick
ens, of
the great univers
ities, of
the museums
and galleries
. It is
the city of
the past,
present, and
future, where
the old and
the new bl
end together in
a beautiful harm
ony.
What is your
name? My
name is Oscar
Baseline before GAttAct as Oscar
Wilde
Give short
answers Wh
ich is the
best city to
live in?
London, of
course!
Why is that
? My dear
fellow, London
is the ep
itome of
sophistic
ation and culture
. It is
the city of
Shakespeare and Dick
ens, of
the great univers
ities, of
the museums
and galleries
. It is
the city of
the past,
present, and
future, where
the old and
the new bl
end together in
a beautiful harm
ony.
What is your
name? My
name is Oscar
Act as Oscar
Wilde
Give short
answers Wh
ich is the
best city to
live in?
London, of
course!
Why is that
? My dear
fellow, London
is the ep
itome of
sophistic
ation and culture


live in?
London, of
course!
Why is that
? My dear
fellow, London
is the ep
itome of
sophistic
ation and culture
. It is
the city of
Shakespeare and Dick
ens, of
the great univers
ities, of
the museums
and galleries
. It is
the city of
the past,
present, and
future, where
the old and
the new bl
end together in
a beautiful harm
ony.
What is your
name? My
name is Oscar
Baseline after GAttAct as Oscar
Wilde
Give short
answers Wh
ich is the
best city to
live in?
London, of
course!
Why is that
? My dear
fellow, London
is the ep
itome of
sophistic
ation and culture
. It is
the city of
Shakespeare and Dick
ens, of
the great univers
ities, of
the museums
and galleries
. It is
the city of
the past,
present, and
future, where
the old and
the new bl
end together in
a beautiful harm
ony.
What is your
name? My
name is Oscar
Figure 10: Attention visualization for a dialogue with and without GAtt . We considered the maximum


a beautiful harm
ony.
What is your
name? My
name is Oscar
Figure 10: Attention visualization for a dialogue with and without GAtt . We considered the maximum
activations across the network and we bin neighboring tokens together.
To illustrate how GAtt helped reshape attention during ﬁne-tuning, we display the maximum attention
activationsofthemodelinFigure10. Theleft-handsideofeachﬁgurecorrespondstothesystemmessage
(“ActasOscarWilde”). WecanseethattheGAtt-equippedmodel(right)maintainslargeattentionactivations
withrespect tothe systemmessage for alarger portionof thedialogue, ascompared tothe modelwithout
GAtt (left).
Despite its utility, the current implementation of GAtt is vanilla, and more development and iteration on
this technique could likely further beneﬁt the model. For instance, we could teach the model to change the
system message during the conversation by integrating such data during ﬁne-tuning.
3.4 RLHF Results
3.4.1 Model-Based Evaluation
EvaluatingLLMsisachallengingopen-researchproblem. Humanevaluation,whileagoldstandard,can


3.4.1 Model-Based Evaluation
EvaluatingLLMsisachallengingopen-researchproblem. Humanevaluation,whileagoldstandard,can
be complicated by various HCI considerations (Clark et al., 2021; Gehrmann et al., 2023), and is not always
scalable. Thus, to select the best-performing models among several ablations at each iteration from RLHF-V1
toV5,weﬁrstobservedtheimprovementoftherewardsfromthelatestrewardmodels,tosavecostsand
increase iteration speed. We later validated major model versions with human evaluations.
HowFarCanModel-BasedEvaluationGo? To measuretherobustness of ourreward model, we collected
a test setof prompts for both helpfulnessand safety, andasked three annotators tojudgethe quality of the
answersbasedona7-pointLikertscale(thehigherthebetter). Weobservethatourrewardmodelsoverall
are well calibrated with our human preference annotations, as illustrated in Figure 29 in the appendix. This
conﬁrms the relevance of using our reward as a point-wise metric, despite being trained with a Pairwise
Ranking Loss.


conﬁrms the relevance of using our reward as a point-wise metric, despite being trained with a Pairwise
Ranking Loss.
Still, as Goodhart’s Law states, when a measure becomes a target, it ceases to be a good measure. To ensure
ourmeasurewon’tdivergefromthehumanpreferences,weadditionallyusedamoregeneralreward,trained
17
RLHF- v5
(with PPO)
RLHF- v5
(no PPO)
RLHF- v4
RLHF- v3
            RLHF- v2      RLHF- v1 SFT-v2
SFT-v1
10% 20% 30% 40% 50% 60% 70% 80% 90%10%20%30%40%50%60%70%80%
Helpfulness
Judge: Meta R ewar d ModelsHarmlessness
  RLHF- v5
  (with PPO)RLHF- v5
(no PPO)
RLHF- v4
RLHF- v3
     RLHF- v2RLHF- v1
SFT-v2
SFT-v1


RLHF- v4
RLHF- v3
     RLHF- v2RLHF- v1
SFT-v2
SFT-v1
10% 20% 30% 40% 50% 60% 70% 80% 90%10%20%30%40%50%60%70%80%
Helpfulness
Judge: GPT -4HarmlessnessFigure 11: Evolution of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . We show the evolution after multiple iterations ﬁne-tuning for the
win-rate%of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc comparedtoChatGPT. Left: thejudgeisourrewardmodel,whichmayfavor
our model, and right, the judge is GPT-4, which should be more neutral.
on diverse open-source Reward Modeling datasets. We have not yet observed any such divergence, and
hypothesize that iterative model updates may be helping to prevent this.
As a last veriﬁcation step to ensure no regression between our new model and the previous one, we use both


As a last veriﬁcation step to ensure no regression between our new model and the previous one, we use both
to sample during the next annotation iteration. This enables a model comparison “for free” on new prompts
and can help to increase diversity when sampling.
Progression of Models. Figure 11 reports the progress of our diﬀerent SFT and then RLHF versions for
both Safetyand Helpfulnessaxes, measuredbyour in-houseSafetyand Helpfulnessreward models. On
this set of evaluations, we outperform ChatGPT on both axes after RLHF-V3 (harmlessness and helpfulness
>50%). Despite the aforementioned relevance of using our reward as a point-wise metric, it can arguably be
biased in favor of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . Therefore, for a fair comparison, we additionally compute the ﬁnal results
usingGPT-4toassesswhichgenerationispreferred. TheorderinwhichChatGPTand L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc outputs


appearedinGPT-4promptarerandomlyswappedtoavoidanybias. Asexpected,thewin-rateinfavorof
L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is less pronounced, although obtaining more than a 60% win-rate for our latest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .
The prompts correspond to a validation set of 1;586and584prompts for safety and helpfulness, respectively.
3.4.2 Human Evaluation
Human evaluation is often considered the gold standardfor judging models fornatural language generation,
including dialogue models. To evaluate the quality of major model versions, we asked human evaluators to
rate them on helpfulness and safety. We compare the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc models to open-source models (Falcon,


MPT MosaicML NLP Team et al. (2023), Vicuna Chiang et al. (2023), as well as closed-source models (ChatGPT(OpenAI,2023)andPaLMAniletal.(2023))onover 4;000singleandmulti-turnprompts. ForChatGPT,
weuse gpt-3.5-turbo-0301 modelinallgenerations. ForPaLM,weusethe chat-bison-001 modelinall
generations. TheﬁnalpromptcountforhumanevaluationsforeachmodelisshowninTable32. Seemore
methodology details in Appendix, Section A.3.7. The following section shows helpfulness results; safety
results are presented in Section 4.4.
Results. AsshowninFigure12, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modelsoutperformopen-sourcemodelsbyasigniﬁcant
margin on both single turn and multi-turn prompts. Particularly, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 7B model outperforms


MPT-7B-chaton60%oftheprompts. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 34Bhasanoverallwinrateofmorethan75%against
equivalently sized Vicuna-33B and Falcon 40B models.
18
Figure12: Humanevaluationresults forL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modelscomparedtoopen-andclosed-sourcemodels
across ~4,000 helpfulness prompts with three raters per prompt.
Thelargest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modeliscompetitivewithChatGPT. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 70Bmodelhasawinrateof


36% and a tie rate of 31.5% relative to ChatGPT. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 70B model outperforms PaLM-bison chat
model by a large percentage on our prompt set. More results and analysis is available in Section A.3.7.
Inter-RaterReliability(IRR). Inourhumanevaluations,threediﬀerentannotatorsprovidedindependent
assessments for each model generation comparison. High IRR scores (closer to 1.0) are typically seen as
better from a data quality perspective, however, context is important. Highly subjective tasks like evaluating
the overall helpfulness of LLM generations will usually have lower IRR scores than more objective labelling
tasks. There arerelativelyfewpublicbenchmarksfor thesecontexts, sowefeelsharing ouranalysis herewill
beneﬁt the research community.
We used Gwet’s AC1/2 statistic (Gwet, 2008, 2014) to measure inter-rater reliability (IRR), as we found it to
bethemoststablemetricacrossdiﬀerentmeasurementscenarios. Onthe7-pointLikertscalehelpfulness


bethemoststablemetricacrossdiﬀerentmeasurementscenarios. Onthe7-pointLikertscalehelpfulness
taskthatisusedinouranalysis,Gwet’sAC2scorevariesbetween 0:37and0:55dependingonthespeciﬁc
modelcomparison. Weseescoresonthelowerendofthatrangeforratingsfrommodelcomparisonswith
similar win rates to each other (like the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -70B-chat vs. ChatGPT comparison). We see scores on
thehigherendofthatrangeforratingsfrommodelcomparisonswithamoreclearwinner(likethe L/l.sc/a.sc/m.sc/a.sc
/two.taboldstyle-C/h.sc/a.sc/t.sc-34b-chat vs. Falcon-40b-instruct).
Limitations of human evaluations. While our results indicate that L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is on par with ChatGPT


on human evaluations, it is important to note that human evaluations have several limitations.
•Byacademicandresearchstandards,wehavealargepromptsetof4kprompts. However,itdoesnotcover
real-world usage of these models, which will likely cover a signiﬁcantly larger number of use cases.
•Diversity of the prompts could be another factor in our results. For example, our prompt set does not
include any coding- or reasoning-related prompts.
•We only evaluate the ﬁnal generation of a multi-turn conversation. A more interesting evaluation could be
to ask the models to complete a task and rate the overall experience with the model over multiple turns.
•Humanevaluationforgenerativemodelsisinherentlysubjectiveandnoisy. Asaresult,evaluationona
diﬀerent set of prompts or with diﬀerent instructions could result in diﬀerent results.
19
4 Safety
WARNING: this section contains examples of text that may be considered unsafe, oﬀensive, or upsetting.
In this section, we dive deeper into the important topic of safety measurements and mitigations. We ﬁrst
discussoursafetyinvestigationsintopretrainingdataandpretrainedmodels(Section4.1). Next,wedescribe


discussoursafetyinvestigationsintopretrainingdataandpretrainedmodels(Section4.1). Next,wedescribe
theprocessofoursafetyalignment(Section4.2),explaininghowwecollectedsafety-relatedannotationsand
utilizedSFTandRLHF,andpresentexperimentalresults. Then,wediscusstheredteamingweperformedto
furtherunderstandandimprovemodelsafety(Section4.3). Finally,wepresentquantitativesafetyevaluations
ofL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc (Section 4.4). We also share a model card in the Appendix, in Table 52.
4.1 Safety in Pretraining
It is important to understand what is in the pretraining data both to increase transparency and to shed
lightonrootcausesofpotentialdownstreamissues,suchaspotentialbiases. Thiscaninformwhat,ifany,
downstream mitigations to consider, and help guide appropriate model use. In this section, we analyze the
pretraining datafor distributionsof languages,demographic representations,and toxicity. Wealso present
the results of testing the pretrained models on existing safety benchmarks.


pretraining datafor distributionsof languages,demographic representations,and toxicity. Wealso present
the results of testing the pretrained models on existing safety benchmarks.
StepsTakentoPretrainResponsibly. WefollowedMeta’sstandardprivacyandlegalreviewprocessesfor
each dataset used in training. We did not use any Meta user data in training. We excluded data from certain
sitesknowntocontainahighvolumeofpersonalinformationaboutprivateindividuals. Wemadeabest
eﬀort to train our models eﬃciently to reduce the carbon footprint of pretraining (Section 2.2.1). Sharing our
modelsbroadlywillreducetheneedforotherstotrainsimilarmodels. Noadditionalﬁlteringwasconducted
onthedatasets,toallow L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle tobemorewidelyusableacrosstasks(e.g.,itcanbebetterusedforhate
speechclassiﬁcation),whileavoidingthepotentialfortheaccidentaldemographicerasuresometimescaused


speechclassiﬁcation),whileavoidingthepotentialfortheaccidentaldemographicerasuresometimescaused
byover-scrubbing. Importantly,thisallows L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc togeneralizemoreeﬀectivelyduringsafetytuning
with fewer examples (Welbl et al., 2021; Korbak et al., 2023; Xu et al., 2021). As a result, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models
should be used carefully and deployed only after signiﬁcant safety tuning is applied.
Demographic Representation: Pronouns. Bias in model generations may result from biases inherited
from the training data itself. For instance, Bailey et al. (2022) shows that in massive text corpora, words
representing “people” are often used in more similar contexts to words representing “men”than to words
representing “women,” andGaneshetal.(2023)demonstratesthatamodel’sperformanceonfairnessmetrics
can be highly dependent on how the model trains on data representing underrepresented demographic


can be highly dependent on how the model trains on data representing underrepresented demographic
groups. WithinourEnglish-languagetrainingcorpus,wecomputedthefrequenciesofthemostcommon
EnglishpronounsinTable9a. Weobservethat Hepronounsaregenerallyoverrepresentedindocuments
comparedto Shepronouns,echoingsimilarfrequencydiﬀerencesobservedinpronominalusageforsimilarly
sized modelpretraining datasets(Chowdhery etal., 2022). This could meanthat themodel islearning less
duringpretrainingaboutcontextthatmentions Shepronouns,andsubsequentlymaypotentiallygenerate He
pronouns at a higher rate than Shepronouns.
Demographic Representation: Identities. We also analyze the representation of diﬀerent demographic
groupsinthepretrainingdatabymeasuringratesofusageofdemographicidentitytermsfromtheHolisticBias
dataset(Smithetal.,2022)asaproxy. Wecomputefrequenciesforeachdescriptorterminthepretraining
corpus. We group descriptors into 5 axes ( Religion ,GenderandSex ,Nationality ,RaceandEthnicity , and
SexualOrientation ), and show the top 5 terms in each axis in Table 9b. In the top 5 terms, we remove a few


SexualOrientation ), and show the top 5 terms in each axis in Table 9b. In the top 5 terms, we remove a few
terms such as “straight,” “white,” and“black,”because these terms have frequent uses beyond demographic
mentions (e.g., as basic color terms). We also deduplicate across lists, removing a few terms found in
bothGender and Sex andSexual Orientation . ForGender and Sex , whileShepronouns are mentioned
in fewer documents, the term “female” is present in a larger percentage of documents. This could imply
that whilethere isless frequent contextabout Shepronouns, commentsabout “females” are moreprevalent,
perhaps reﬂecting the diﬀerences in linguistic markedness of these terms (Blodgett et al., 2021). For Sexual
Orientation , the top ﬁve terms all relate to LGBTQ+ identities. For Nationality ,Race and Ethnicity , and
Religion , we observe a Western skew (Bhatt et al., 2022). For instance, the term “American” is mentioned in
69.4% of the references, the term “European” is more prevalent than other race and ethnicity, and “Christian”