Impact of Computation in Integral Reinforcement Learning for
  Continuous-Time Control
Wenhan Cao Wei Pan
Integral reinforcement learning (IntRL) demands the precise computation of
the utility function's integral at its policy evaluation (PEV) stage. This is
achieved through quadrature rules, which are weighted sums of utility functions
evaluated from state samples obtained in discrete time. Our research reveals a
critical yet underexplored phenomenon: the choice of the computational method
-- in this case, the quadrature rule -- can significantly impact control
performance. This impact is traced back to the fact that computational errors
introduced in the PEV stage can affect the policy iteration's convergence
behavior, which in turn affects the learned controller. To elucidate how
computation impacts control, we draw a parallel between IntRL's policy
iteration and Newton's method applied to the Hamilton-Jacobi-Bellman equation.
In this light, computational error in PEV manifests as an extra error term in
each iteration of Newton's method, with its upper bound proportional to the
computational error. Further, we demonstrate that when the utility function
resides in a reproducing kernel Hilbert space (RKHS), the optimal quadrature is
achievable by employing Bayesian quadrature with the RKHS-inducing kernel
function. We prove that the local convergence rates for IntRL using the
trapezoidal rule and Bayesian quadrature with a Mat\'ern kernel to be
$O(N^{-2})$ and $O(N^{-b})$, where $N$ is the number of evenly-spaced samples
and $b$ is the Mat\'ern kernel's smoothness parameter. These theoretical
findings are finally validated by two canonical control tasks.