Actor-Critic Reinforcement Learning with Phased Actor
Ruofan Wu Junmin Zhong Jennie Si
Policy gradient methods in actor-critic reinforcement learning (RL) have
become perhaps the most promising approaches to solving continuous optimal
control problems. However, the trial-and-error nature of RL and the inherent
randomness associated with solution approximations cause variations in the
learned optimal values and policies. This has significantly hindered their
successful deployment in real life applications where control responses need to
meet dynamic performance criteria deterministically. Here we propose a novel
phased actor in actor-critic (PAAC) method, aiming at improving policy gradient
estimation and thus the quality of the control policy. Specifically, PAAC
accounts for both $Q$ value and TD error in its actor update. We prove
qualitative properties of PAAC for learning convergence of the value and
policy, solution optimality, and stability of system dynamics. Additionally, we
show variance reduction in policy gradient estimation. PAAC performance is
systematically and quantitatively evaluated in this study using DeepMind
Control Suite (DMC). Results show that PAAC leads to significant performance
improvement measured by total cost, learning variance, robustness, learning
speed and success rate. As PAAC can be piggybacked onto general policy gradient
learning frameworks, we select well-known methods such as direct heuristic
dynamic programming (dHDP), deep deterministic policy gradient (DDPG) and their
variants to demonstrate the effectiveness of PAAC. Consequently we provide a
unified view on these related policy gradient algorithms.