DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like
  Models at All Scales
Zhewei Yao Reza Yazdani Aminabadi Olatunji Ruwase Samyam Rajbhandari Xiaoxia Wu Ammar Ahmad Awan Jeff Rasley Minjia Zhang Conglong Li Connor Holmes Zhongzhu Zhou Michael Wyatt Molly Smith Lev Kurilenko Heyang Qin Masahiro Tanaka Shuai Che Shuaiwen Leon Song Yuxiong He
ChatGPT-like models have revolutionized various applications in artificial
intelligence, from summarization and coding to translation, matching or even
surpassing human performance. However, the current landscape lacks an
accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement
Learning with Human Feedback) training pipeline for these powerful models,
particularly when training at the scale of billions of parameters. This paper
introduces DeepSpeed-Chat, a novel system that democratizes RLHF training,
making it accessible to the AI community. DeepSpeed-Chat offers three key
capabilities: an easy-to-use training and inference experience for ChatGPT-like
models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from
InstructGPT, and a robust DeepSpeed-RLHF system that combines various
optimizations for training and inference in a unified way. The system delivers
unparalleled efficiency and scalability, enabling training of models with
hundreds of billions of parameters in record time and at a fraction of the
cost. With this development, DeepSpeed-Chat paves the way for broader access to
advanced RLHF training, even for data scientists with limited resources,
thereby fostering innovation and further development in the field of AI.