Language-Guided World Models: A Model-Based Approach to AI Control
Alex Zhang Khanh Nguyen Jens Tuyls Albert Lin Karthik Narasimhan
Installing probabilistic world models into artificial agents opens an
efficient channel for humans to communicate with and control these agents. In
addition to updating agent policies, humans can modify their internal world
models in order to influence their decisions. The challenge, however, is that
currently existing world models are difficult for humans to adapt because they
lack a natural communication interface. Aimed at addressing this shortcoming,
we develop Language-Guided World Models (LWMs), which can capture environment
dynamics by reading language descriptions. These models enhance agent
communication efficiency, allowing humans to simultaneously alter their
behavior on multiple tasks with concise language feedback. They also enable
agents to self-learn from texts originally written to instruct humans. To
facilitate the development of LWMs, we design a challenging benchmark based on
the game of MESSENGER (Hanjie et al., 2021), requiring compositional
generalization to new language descriptions and environment dynamics. Our
experiments reveal that the current state-of-the-art Transformer architecture
performs poorly on this benchmark, motivating us to design a more robust
architecture. To showcase the practicality of our proposed LWMs, we simulate a
scenario where these models augment the interpretability and safety of an agent
by enabling it to generate and discuss plans with a human before execution. By
effectively incorporating language feedback on the plan, the models boost the
agent performance in the real environment by up to three times without
collecting any interactive experiences in this environment.