ScreenAgent: A Vision Language Model-driven Computer Control Agent
Runliang Niu Jindong Li Shiqi Wang Yali Fu Xiyu Hu Xueyuan Leng He Kong Yi Chang Qi Wang
Existing Large Language Models (LLM) can invoke a variety of tools and APIs
to complete complex tasks. The computer, as the most powerful and universal
tool, could potentially be controlled directly by a trained LLM agent. Powered
by the computer, we can hopefully build a more generalized agent to assist
humans in various daily digital works. In this paper, we construct an
environment for a Vision Language Model (VLM) agent to interact with a real
computer screen. Within this environment, the agent can observe screenshots and
manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard
actions. We also design an automated control pipeline that includes planning,
acting, and reflecting phases, guiding the agent to continuously interact with
the environment and complete multi-step tasks. Additionally, we construct the
ScreenAgent Dataset, which collects screenshots and action sequences when
completing a variety of daily computer tasks. Finally, we trained a model,
ScreenAgent, which achieved computer control capabilities comparable to GPT-4V
and demonstrated more precise UI positioning capabilities. Our attempts could
inspire further research on building a generalist LLM agent. The code is
available at \url{https://github.com/niuzaisheng/ScreenAgent}.