Skill Transfer and Discovery for Sim-to-Real Learning: A
  Representation-Based Viewpoint
Haitong Ma Zhaolin Ren Bo Dai Na Li
We study sim-to-real skill transfer and discovery in the context of robotics
control using representation learning. We draw inspiration from spectral
decomposition of Markov decision processes. The spectral decomposition brings
about representation that can linearly represent the state-action value
function induced by any policies, thus can be regarded as skills. The skill
representations are transferable across arbitrary tasks with the same
transition dynamics. Moreover, to handle the sim-to-real gap in the dynamics,
we propose a skill discovery algorithm that learns new skills caused by the
sim-to-real gap from real-world data. We promote the discovery of new skills by
enforcing orthogonal constraints between the skills to learn and the skills
from simulators, and then synthesize the policy using the enlarged skill sets.
We demonstrate our methodology by transferring quadrotor controllers from
simulators to Crazyflie 2.1 quadrotors. We show that we can learn the skill
representations from a single simulator task and transfer these to multiple
different real-world tasks including hovering, taking off, landing and
trajectory tracking. Our skill discovery approach helps narrow the sim-to-real
gap and improve the real-world controller performance by up to 30.2%.