On the Multi-turn Instruction Following for Conversational Web Agents
Yang Deng Xuan Zhang Wenxuan Zhang Yifei Yuan See-Kiong Ng Tat-Seng Chua
Web agents powered by Large Language Models (LLMs) have demonstrated
remarkable abilities in planning and executing multi-step interactions within
complex web-based environments, fulfilling a wide range of web navigation
tasks. Despite these advancements, the potential for LLM-powered agents to
effectively engage with sequential user instructions in real-world scenarios
has not been fully explored. In this work, we introduce a new task of
Conversational Web Navigation, which necessitates sophisticated interactions
that span multiple turns with both the users and the environment, supported by
a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To
tackle the limited context length of LLMs and the context-dependency issue of
the conversational tasks, we further propose a novel framework, named
self-reflective memory-augmented planning (Self-MAP), which employs memory
utilization and self-reflection techniques. Extensive experiments are conducted
to benchmark the MT-Mind2Web dataset, and validate the effectiveness of the
proposed method.