MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot
  Learning
Rafael Rafailov Kyle Hatch Victor Kolev John D. Martin Mariano Phielipp Chelsea Finn
We study the problem of offline pre-training and online fine-tuning for
reinforcement learning from high-dimensional observations in the context of
realistic robot tasks. Recent offline model-free approaches successfully use
online fine-tuning to either improve the performance of the agent over the data
collection policy or adapt to novel tasks. At the same time, model-based RL
algorithms have achieved significant progress in sample efficiency and the
complexity of the tasks they can solve, yet remain under-utilized in the
fine-tuning setting. In this work, we argue that existing model-based offline
RL methods are not suitable for offline-to-online fine-tuning in
high-dimensional domains due to issues with distribution shifts, off-dynamics
data, and non-stationary rewards. We propose an on-policy model-based method
that can efficiently reuse prior data through model-based value expansion and
policy regularization, while preventing model exploitation by controlling
epistemic uncertainty. We find that our approach successfully solves tasks from
the MetaWorld benchmark, as well as the Franka Kitchen robot manipulation
environment completely from images. To the best of our knowledge, MOTO is the
first method to solve this environment from pixels.