Multimodal Embodied Interactive Agent for Cafe Scene
Yang Liu Xinshuai Song Kaixuan Jiang Weixing Chen Jingzhou Luo Guanbin Li Liang Lin
With the surge in the development of large language models, embodied
intelligence has attracted increasing attention. Nevertheless, prior works on
embodied intelligence typically encode scene or historical memory in an
unimodal manner, either visual or linguistic, which complicates the alignment
of the model's action planning with embodied control. To overcome this
limitation, we introduce the Multimodal Embodied Interactive Agent (MEIA),
capable of translating high-level tasks expressed in natural language into a
sequence of executable actions. Specifically, we propose a novel Multimodal
Environment Memory (MEM) module, facilitating the integration of embodied
control with large models through the visual-language memory of scenes. This
capability enables MEIA to generate executable action plans based on diverse
requirements and the robot's capabilities. We conduct experiments in a dynamic
virtual cafe environment, utilizing multiple large models through zero-shot
learning, and carefully design scenarios for various situations. The
experimental results showcase the promising performance of our MEIA in various
embodied interactive tasks.