Language Model Self-improvement by Reinforcement Learning Contemplation
Jing-Cheng Pang Pengyuan Wang Kaiyuan Li Xiong-Hui Chen Jiacheng Xu Zongzhang Zhang Yang Yu
Large Language Models (LLMs) have exhibited remarkable performance across
various natural language processing (NLP) tasks. However, fine-tuning these
models often necessitates substantial supervision, which can be expensive and
time-consuming to obtain. This paper introduces a novel unsupervised method
called LanguageModel Self-Improvement by Reinforcement Learning Contemplation
(SIRLC) that improves LLMs without reliance on external labels. Our approach is
grounded in the observation that it is simpler for language models to assess
text quality than to generate text. Building on this insight, SIRLC assigns
LLMs dual roles as both student and teacher. As a student, the LLM generates
answers to unlabeled questions, while as a teacher, it evaluates the generated
text and assigns scores accordingly. The model parameters are updated using
reinforcement learning to maximize the evaluation score. We demonstrate that
SIRLC can be applied to various NLP tasks, such as reasoning problems, text
generation, and machine translation. Our experiments show that SIRLC
effectively improves LLM performance without external supervision, resulting in
a 5.6% increase in answering accuracy for reasoning tasks and a rise in
BERTScore from 0.82 to 0.86 for translation tasks. Furthermore, SIRLC can be
applied to models of different sizes, showcasing its broad applicability.