TWIST: Teacher-Student World Model Distillation for Efficient
  Sim-to-Real Transfer
Jun Yamada Marc Rigter Jack Collins Ingmar Posner
Model-based RL is a promising approach for real-world robotics due to its
improved sample efficiency and generalization capabilities compared to
model-free RL. However, effective model-based RL solutions for vision-based
real-world applications require bridging the sim-to-real gap for any world
model learnt. Due to its significant computational cost, standard domain
randomisation does not provide an effective solution to this problem. This
paper proposes TWIST (Teacher-Student World Model Distillation for Sim-to-Real
Transfer) to achieve efficient sim-to-real transfer of vision-based model-based
RL using distillation. Specifically, TWIST leverages state observations as
readily accessible, privileged information commonly garnered from a simulator
to significantly accelerate sim-to-real transfer. Specifically, a teacher world
model is trained efficiently on state information. At the same time, a matching
dataset is collected of domain-randomised image observations. The teacher world
model then supervises a student world model that takes the domain-randomised
image observations as input. By distilling the learned latent dynamics model
from the teacher to the student model, TWIST achieves efficient and effective
sim-to-real transfer for vision-based model-based RL tasks. Experiments in
simulated and real robotics tasks demonstrate that our approach outperforms
naive domain randomisation and model-free methods in terms of sample efficiency
and task performance of sim-to-real transfer.