ICED: Zero-Shot Transfer in Reinforcement Learning via In-Context
  Environment Design
Samuel Garcin James Doran Shangmin Guo Christopher G. Lucas Stefano V. Albrecht
Autonomous agents trained using deep reinforcement learning (RL) often lack
the ability to successfully generalise to new environments, even when they
share characteristics with the environments they have encountered during
training. In this work, we investigate how the sampling of individual
environment instances, or levels, affects the zero-shot generalisation (ZSG)
ability of RL agents. We discover that, for deep actor-critic architectures
sharing their base layers, prioritising levels according to their value loss
minimises the mutual information between the agent's internal representation
and the set of training levels in the generated training data. This provides a
novel theoretical justification for the implicit regularisation achieved by
certain adaptive sampling strategies. We then turn our attention to
unsupervised environment design (UED) methods, which have more control over the
data generation mechanism. We find that existing UED methods can significantly
shift the training distribution, which translates to low ZSG performance. To
prevent both overfitting and distributional shift, we introduce in-context
environment design (ICED). ICED generates levels using a variational
autoencoder trained over an initial set of level parameters, reducing
distributional shift, and achieves significant improvements in ZSG over
adaptive level sampling strategies and UED methods.