On the Effectiveness of Offline RL for Dialogue Response Generation
Paloma Sodhi Felix Wu Ethan R. Elenberg Kilian Q. Weinberger Ryan McDonald
A common training technique for language models is teacher forcing (TF). TF
attempts to match human language exactly, even though identical meanings can be
expressed in different ways. This motivates use of sequence-level objectives
for dialogue response generation. In this paper, we study the efficacy of
various offline reinforcement learning (RL) methods to maximize such
objectives. We present a comprehensive evaluation across multiple datasets,
models, and metrics. Offline RL shows a clear performance improvement over
teacher forcing while not inducing training instability or sacrificing
practical training budgets.