Stepwise Alignment for Constrained Language Model Policy Optimization
Akifumi Wachi Thien Q Tran Rei Sato Takumi Tanabe Yohei Akimoto
Safety and trustworthiness are indispensable requirements for applying AI
systems based on large language models (LLMs) in real-world applications. This
paper formulates a human value alignment as a language model policy
optimization problem to maximize reward under a safety constraint and then
proposes an algorithm called Stepwise Alignment for Constrained Policy
Optimization (SACPO). A key idea behind SACPO, supported by theory, is that the
optimal policy incorporating both reward and safety can be directly obtained
from a reward-aligned policy. Based on this key idea, SACPO aligns the LLMs
with each metric step-wise while leveraging simple yet powerful alignment
algorithms such as direct preference optimization (DPO). SACPO provides many
benefits such as simplicity, stability, computational efficiency, and
flexibility regarding algorithms and dataset selection. Under mild assumption,
our theoretical analysis provides the upper bounds regarding near-optimality
and safety constraint violation. Our experimental results show that SACPO can
fine-tune Alpaca-7B better than the state-of-the-art method in terms of both
helpfulness and harmlessness