Decoupling Meta-Reinforcement Learning with Gaussian Task Contexts and
  Skills
Hongcai He Anjie Zhu Shuang Liang Feiyu Chen Jie Shao
Offline meta-reinforcement learning (meta-RL) methods, which adapt to unseen
target tasks with prior experience, are essential in robot control tasks.
Current methods typically utilize task contexts and skills as prior experience,
where task contexts are related to the information within each task and skills
represent a set of temporally extended actions for solving subtasks. However,
these methods still suffer from limited performance when adapting to unseen
target tasks, mainly because the learned prior experience lacks generalization,
i.e., they are unable to extract effective prior experience from meta-training
tasks by exploration and learning of continuous latent spaces. We propose a
framework called decoupled meta-reinforcement learning (DCMRL), which (1)
contrastively restricts the learning of task contexts through pulling in
similar task contexts within the same task and pushing away different task
contexts of different tasks, and (2) utilizes a Gaussian quantization
variational autoencoder (GQ-VAE) for clustering the Gaussian distributions of
the task contexts and skills respectively, and decoupling the exploration and
learning processes of their spaces. These cluster centers which serve as
representative and discrete distributions of task context and skill are stored
in task context codebook and skill codebook, respectively. DCMRL can acquire
generalizable prior experience and achieve effective adaptation to unseen
target tasks during the meta-testing phase. Experiments in the navigation and
robot manipulation continuous control tasks show that DCMRL is more effective
than previous meta-RL methods with more generalizable prior experience.