Deep Model Predictive Optimization
Jacob Sacks Rwik Rana Kevin Huang Alex Spitzer Guanya Shi Byron Boots
A major challenge in robotics is to design robust policies which enable
complex and agile behaviors in the real world. On one end of the spectrum, we
have model-free reinforcement learning (MFRL), which is incredibly flexible and
general but often results in brittle policies. In contrast, model predictive
control (MPC) continually re-plans at each time step to remain robust to
perturbations and model inaccuracies. However, despite its real-world
successes, MPC often under-performs the optimal strategy. This is due to model
quality, myopic behavior from short planning horizons, and approximations due
to computational constraints. And even with a perfect model and enough compute,
MPC can get stuck in bad local optima, depending heavily on the quality of the
optimization algorithm. To this end, we propose Deep Model Predictive
Optimization (DMPO), which learns the inner-loop of an MPC optimization
algorithm directly via experience, specifically tailored to the needs of the
control problem. We evaluate DMPO on a real quadrotor agile trajectory tracking
task, on which it improves performance over a baseline MPC algorithm for a
given computational budget. It can outperform the best MPC algorithm by up to
27% with fewer samples and an end-to-end policy trained with MFRL by 19%.
Moreover, because DMPO requires fewer samples, it can also achieve these
benefits with 4.3X less memory. When we subject the quadrotor to turbulent wind
fields with an attached drag plate, DMPO can adapt zero-shot while still
outperforming all baselines. Additional results can be found at
https://tinyurl.com/mr2ywmnw.