Training a Vision Language Model as Smartphone Assistant
Nicolai Dorka Janusz Marecki Ammar Anwar
Addressing the challenge of a digital assistant capable of executing a wide
array of user tasks, our research focuses on the realm of instruction-based
mobile device control. We leverage recent advancements in large language models
(LLMs) and present a visual language model (VLM) that can fulfill diverse tasks
on mobile devices. Our model functions by interacting solely with the user
interface (UI). It uses the visual input from the device screen and mimics
human-like interactions, encompassing gestures such as tapping and swiping.
This generality in the input and output space allows our agent to interact with
any application on the device. Unlike previous methods, our model operates not
only on a single screen image but on vision-language sentences created from
sequences of past screenshots along with corresponding actions. Evaluating our
method on the challenging Android in the Wild benchmark demonstrates its
promising efficacy and potential.