Clipped-Objective Policy Gradients for Pessimistic Policy Optimization
Jared Markowitz Edward W. Staley
To facilitate efficient learning, policy gradient approaches to deep
reinforcement learning (RL) are typically paired with variance reduction
measures and strategies for making large but safe policy changes based on a
batch of experiences. Natural policy gradient methods, including Trust Region
Policy Optimization (TRPO), seek to produce monotonic improvement through
bounded changes in policy outputs. Proximal Policy Optimization (PPO) is a
commonly used, first-order algorithm that instead uses loss clipping to take
multiple safe optimization steps per batch of data, replacing the bound on the
single step of TRPO with regularization on multiple steps. In this work, we
find that the performance of PPO, when applied to continuous action spaces, may
be consistently improved through a simple change in objective. Instead of the
importance sampling objective of PPO, we instead recommend a basic policy
gradient, clipped in an equivalent fashion. While both objectives produce
biased gradient estimates with respect to the RL objective, they also both
display significantly reduced variance compared to the unbiased off-policy
policy gradient. Additionally, we show that (1) the clipped-objective policy
gradient (COPG) objective is on average "pessimistic" compared to both the PPO
objective and (2) this pessimism promotes enhanced exploration. As a result, we
empirically observe that COPG produces improved learning compared to PPO in
single-task, constrained, and multi-task learning, without adding significant
computational cost or complexity. Compared to TRPO, the COPG approach is seen
to offer comparable or superior performance, while retaining the simplicity of
a first-order method.