On-Policy Policy Gradient Reinforcement Learning Without On-Policy
  Sampling
Nicholas E. Corrado Josiah P. Hanna
On-policy reinforcement learning (RL) algorithms perform policy updates using
i.i.d. trajectories collected by the current policy. However, after observing
only a finite number of trajectories, on-policy sampling may produce data that
fails to match the expected on-policy data distribution. This sampling error
leads to noisy updates and data inefficient on-policy learning. Recent work in
the policy evaluation setting has shown that non-i.i.d., off-policy sampling
can produce data with lower sampling error than on-policy sampling can produce.
Motivated by this observation, we introduce an adaptive, off-policy sampling
method to improve the data efficiency of on-policy policy gradient algorithms.
Our method, Proximal Robust On-Policy Sampling (PROPS), reduces sampling error
by collecting data with a behavior policy that increases the probability of
sampling actions that are under-sampled with respect to the current policy.
Rather than discarding data from old policies -- as is commonly done in
on-policy algorithms -- PROPS uses data collection to adjust the distribution
of previously collected data to be approximately on-policy. We empirically
evaluate PROPS on both continuous-action MuJoCo benchmark tasks as well as
discrete-action tasks and demonstrate that (1) PROPS decreases sampling error
throughout training and (2) improves the data efficiency of on-policy policy
gradient algorithms. Our work improves the RL community's understanding of a
nuance in the on-policy vs off-policy dichotomy: on-policy learning requires
on-policy data, not on-policy sampling.