Signatures Meet Dynamic Programming: Generalizing Bellman Equations for
  Trajectory Following
Motoya Ohnishi Iretiayo Akinola Jie Xu Ajay Mandlekar Fabio Ramos
Path signatures have been proposed as a powerful representation of paths that
efficiently captures the path's analytic and geometric characteristics, having
useful algebraic properties including fast concatenation of paths through
tensor products. Signatures have recently been widely adopted in machine
learning problems for time series analysis. In this work we establish
connections between value functions typically used in optimal control and
intriguing properties of path signatures. These connections motivate our novel
control framework with signature transforms that efficiently generalizes the
Bellman equation to the space of trajectories. We analyze the properties and
advantages of the framework, termed signature control. In particular, we
demonstrate that (i) it can naturally deal with varying/adaptive time steps;
(ii) it propagates higher-level information more efficiently than value
function updates; (iii) it is robust to dynamical system misspecification over
long rollouts. As a specific case of our framework, we devise a model
predictive control method for path tracking. This method generalizes integral
control, being suitable for problems with unknown disturbances. The proposed
algorithms are tested in simulation, with differentiable physics models
including typical control and robotics tasks such as point-mass, curve
following for an ant model, and a robotic manipulator.