Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot
  Generalization
Sai Prasanna Karim Farid Raghu Rajan André Biedenkapp
Zero-shot generalization (ZSG) to unseen dynamics is a major challenge for
creating generally capable embodied agents. To address the broader challenge,
we start with the simpler setting of contextual reinforcement learning (cRL),
assuming observability of the context values that parameterize the variation in
the system's dynamics, such as the mass or dimensions of a robot, without
making further simplifying assumptions about the observability of the Markovian
state. Toward the goal of ZSG to unseen variation in context, we propose the
contextual recurrent state-space model (cRSSM), which introduces changes to the
world model of the Dreamer (v3) (Hafner et al., 2023). This allows the world
model to incorporate context for inferring latent Markovian states from the
observations and modeling the latent dynamics. Our experiments show that such
systematic incorporation of the context improves the ZSG of the policies
trained on the ``dreams'' of the world model. We further find qualitatively
that our approach allows Dreamer to disentangle the latent state from context,
allowing it to extrapolate its dreams to the many worlds of unseen contexts.
The code for all our experiments is available at
\url{https://github.com/sai-prasanna/dreaming_of_many_worlds}.