WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work
  Tasks?
Alexandre Drouin Maxime Gasse Massimo Caccia Issam H. Laradji Manuel Del Verme Tom Marty Léo Boisvert Megh Thakkar Quentin Cappart David Vazquez Nicolas Chapados Alexandre Lacoste
We study the use of large language model-based agents for interacting with
software via web browsers. Unlike prior work, we focus on measuring the agents'
ability to perform tasks that span the typical daily work of knowledge workers
utilizing enterprise software systems. To this end, we propose WorkArena, a
remote-hosted benchmark of 29 tasks based on the widely-used ServiceNow
platform. We also introduce BrowserGym, an environment for the design and
evaluation of such agents, offering a rich set of actions as well as multimodal
observations. Our empirical evaluation reveals that while current agents show
promise on WorkArena, there remains a considerable gap towards achieving full
task automation. Notably, our analysis uncovers a significant performance
disparity between open and closed-source LLMs, highlighting a critical area for
future exploration and development in the field.