Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions
Hui Yang Sifu Yue Yunzhong He
Auto-GPT is an autonomous agent that leverages recent advancements in
adapting Large Language Models (LLMs) for decision-making tasks. While there
has been a growing interest in Auto-GPT stypled agents, questions remain
regarding the effectiveness and flexibility of Auto-GPT in solving real-world
decision-making tasks. Its limited capability for real-world engagement and the
absence of benchmarks contribute to these uncertainties. In this paper, we
present a comprehensive benchmark study of Auto-GPT styled agents in
decision-making tasks that simulate real-world scenarios. Our aim is to gain
deeper insights into this problem and understand the adaptability of GPT-based
agents. We compare the performance of popular LLMs such as GPT-4, GPT-3.5,
Claude, and Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we
introduce the Additional Opinions algorithm, an easy and effective method that
incorporates supervised/imitation-based learners into the Auto-GPT scheme. This
approach enables lightweight supervised learning without requiring fine-tuning
of the foundational LLMs. We demonstrate through careful baseline comparisons
and ablation studies that the Additional Opinions algorithm significantly
enhances performance in online decision-making benchmarks, including WebShop
and ALFWorld.