Simplified Temporal Consistency Reinforcement Learning
Yi Zhao Wenshuai Zhao Rinu Boney Juho Kannala Joni Pajarinen
Reinforcement learning is able to solve complex sequential decision-making
tasks but is currently limited by sample efficiency and required computation.
To improve sample efficiency, recent work focuses on model-based RL which
interleaves model learning with planning. Recent methods further utilize policy
learning, value estimation, and, self-supervised learning as auxiliary
objectives. In this paper we show that, surprisingly, a simple representation
learning approach relying only on a latent dynamics model trained by latent
temporal consistency is sufficient for high-performance RL. This applies when
using pure planning with a dynamics model conditioned on the representation,
but, also when utilizing the representation as policy and value function
features in model-free RL. In experiments, our approach learns an accurate
dynamics model to solve challenging high-dimensional locomotion tasks with
online planners while being 4.1 times faster to train compared to
ensemble-based methods. With model-free RL without planning, especially on
high-dimensional tasks, such as the DeepMind Control Suite Humanoid and Dog
tasks, our approach outperforms model-free methods by a large margin and
matches model-based methods' sample efficiency while training 2.4 times faster.