Motif: Intrinsic Motivation from Artificial Intelligence Feedback
Martin Klissarov Pierluca D'Oro Shagun Sodhani Roberta Raileanu Pierre-Luc Bacon Pascal Vincent Amy Zhang Mikael Henaff
Exploring rich environments and evaluating one's actions without prior
knowledge is immensely challenging. In this paper, we propose Motif, a general
method to interface such prior knowledge from a Large Language Model (LLM) with
an agent. Motif is based on the idea of grounding LLMs for decision-making
without requiring them to interact with the environment: it elicits preferences
from an LLM over pairs of captions to construct an intrinsic reward, which is
then used to train agents with reinforcement learning. We evaluate Motif's
performance and behavior on the challenging, open-ended and
procedurally-generated NetHack game. Surprisingly, by only learning to maximize
its intrinsic reward, Motif achieves a higher game score than an algorithm
directly trained to maximize the score itself. When combining Motif's intrinsic
reward with the environment reward, our method significantly outperforms
existing approaches and makes progress on tasks where no advancements have ever
been made without demonstrations. Finally, we show that Motif mostly generates
intuitive human-aligned behaviors which can be steered easily through prompt
modifications, while scaling well with the LLM size and the amount of
information given in the prompt.