Creative Agents: Empowering Agents with Imagination for Creative Tasks
Chi Zhang Penglin Cai Yuhui Fu Haoqi Yuan Zongqing Lu
We study building embodied agents for open-ended creative tasks. While
existing methods build instruction-following agents that can perform diverse
open-ended tasks, none of them demonstrates creativity -- the ability to give
novel and diverse task solutions implicit in the language instructions. This
limitation comes from their inability to convert abstract language instructions
into concrete task goals in the environment and perform long-horizon planning
for such complicated goals. Given the observation that humans perform creative
tasks with the help of imagination, we propose a class of solutions for
creative agents, where the controller is enhanced with an imaginator that
generates detailed imaginations of task outcomes conditioned on language
instructions. We introduce several approaches to implementing the components of
creative agents. We implement the imaginator with either a large language model
for textual imagination or a diffusion model for visual imagination. The
controller can either be a behavior-cloning policy learned from data or a
pre-trained foundation model generating executable codes in the environment. We
benchmark creative tasks with the challenging open-world game Minecraft, where
the agents are asked to create diverse buildings given free-form language
instructions. In addition, we propose novel evaluation metrics for open-ended
creative tasks utilizing GPT-4V, which holds many advantages over existing
metrics. We perform a detailed experimental analysis of creative agents,
showing that creative agents are the first AI agents accomplishing diverse
building creation in the survival mode of Minecraft. Our benchmark and models
are open-source for future research on creative agents
(https://github.com/PKU-RL/Creative-Agents).