AVA: Towards Autonomous Visualization Agents through Visual
  Perception-Driven Decision-Making
Shusen Liu Haichao Miao Zhimin Li Matthew Olson Valerio Pascucci Peer-Timo Bremer
With recent advances in multi-modal foundation models, the previously
text-only large language models (LLM) have evolved to incorporate visual input,
opening up unprecedented opportunities for various applications in
visualization. Our work explores the utilization of the visual perception
ability of multi-modal LLMs to develop Autonomous Visualization Agents (AVAs)
that can interpret and accomplish user-defined visualization objectives through
natural language. We propose the first framework for the design of AVAs and
present several usage scenarios intended to demonstrate the general
applicability of the proposed paradigm. The addition of visual perception
allows AVAs to act as the virtual visualization assistant for domain experts
who may lack the knowledge or expertise in fine-tuning visualization outputs.
Our preliminary exploration and proof-of-concept agents suggest that this
approach can be widely applicable whenever the choices of appropriate
visualization parameters require the interpretation of previous visual output.
Feedback from unstructured interviews with experts in AI research, medical
visualization, and radiology has been incorporated, highlighting the
practicality and potential of AVAs. Our study indicates that AVAs represent a
general paradigm for designing intelligent visualization systems that can
achieve high-level visualization goals, which pave the way for developing
expert-level visualization agents in the future.