Deep Metric Tensor Regularized Policy Gradient
Gang Chen Victoria Huang
Policy gradient algorithms are an important family of deep reinforcement
learning techniques. Many past research endeavors focused on using the
first-order policy gradient information to train policy networks. Different
from these works, we conduct research in this paper driven by the believe that
properly utilizing and controlling Hessian information associated with the
policy gradient can noticeably improve the performance of policy gradient
algorithms. One key Hessian information that attracted our attention is the
Hessian trace, which gives the divergence of the policy gradient vector field
in the Euclidean policy parametric space. We set the goal to generalize this
Euclidean policy parametric space into a general Riemmanian manifold by
introducing a metric tensor field $g_ab$ in the parametric space. This is
achieved through newly developed mathematical tools, deep learning algorithms,
and metric tensor deep neural networks (DNNs). Armed with these technical
developments, we propose a new policy gradient algorithm that learns to
minimize the absolute divergence in the Riemannian manifold as an important
regularization mechanism, allowing the Riemannian manifold to smoothen its
policy gradient vector field. The newly developed algorithm is experimentally
studied on several benchmark reinforcement learning problems. Our experiments
clearly show that the new metric tensor regularized algorithm can significantly
outperform its counterpart that does not use our regularization technique.
Additional experimental analysis further suggests that the trained metric
tensor DNN and the corresponding metric tensor $g_{ab}$ can effectively reduce
the absolute divergence towards zero in the Riemannian manifold.