Model-Free $δ$-Policy Iteration Based on Damped Newton Method for
  Nonlinear Continuous-Time H$\infty$ Tracking Control
Qi Wang
This paper presents a {\delta}-PI algorithm which is based on damped Newton
method for the H{\infty} tracking control problem of unknown continuous-time
nonlinear system. A discounted performance function and an augmented system are
used to get the tracking Hamilton-Jacobi-Isaac (HJI) equation. Tracking HJI
equation is a nonlinear partial differential equation, traditional
reinforcement learning methods for solving the tracking HJI equation are mostly
based on the Newton method, which usually only satisfies local convergence and
needs a good initial guess. Based upon the damped Newton iteration operator
equation, a generalized tracking Bellman equation is derived firstly. The
{\delta}-PI algorithm can seek the optimal solution of the tracking HJI
equation by iteratively solving the generalized tracking Bellman equation.
On-policy learning and off-policy learning {\delta}-PI reinforcement learning
methods are provided, respectively. Off-policy version {\delta}-PI algorithm is
a model-free algorithm which can be performed without making use of a priori
knowledge of the system dynamics. NN-based implementation scheme for the
off-policy {\delta}-PI algorithms is shown. The suitability of the model-free
{\delta}-PI algorithm is illustrated with a nonlinear system simulation.