WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents
Michael Lutz Arth Bohra Manvel Saroyan Artem Harutyunyan Giovanni Campagna
In the realm of web agent research, achieving both generalization and
accuracy remains a challenging problem. Due to high variance in website
structure, existing approaches often fail. Moreover, existing fine-tuning and
in-context learning techniques fail to generalize across multiple websites. We
introduce Wilbur, an approach that uses a differentiable ranking model and a
novel instruction synthesis technique to optimally populate a black-box large
language model's prompt with task demonstrations from previous runs. To
maximize end-to-end success rates, we also propose an intelligent backtracking
mechanism that learns and recovers from its mistakes. Finally, we show that our
ranking model can be trained on data from a generative auto-curriculum which
samples representative goals from an LLM, runs the agent, and automatically
evaluates it, with no manual annotation. Wilbur achieves state-of-the-art
results on the WebVoyager benchmark, beating text-only models by 8% overall,
and up to 36% on certain websites. On the same benchmark, Wilbur is within 5%
of a strong multi-modal model despite only receiving textual inputs, and
further analysis reveals a substantial number of failures are due to
engineering challenges of operating the web.