AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making
This addresses the need for domain experts who lack visualization expertise to fine-tune outputs, though it appears incremental as it builds on existing multi-modal LLMs.
The authors tackled the problem of automating visualization tasks by developing Autonomous Visualization Agents (AVAs) that use multi-modal LLMs with visual perception to interpret and achieve user-defined objectives through natural language, presenting a framework and proof-of-concept agents with feedback from experts in AI, medical visualization, and radiology.
With recent advances in multi-modal foundation models, the previously text-only large language models (LLM) have evolved to incorporate visual input, opening up unprecedented opportunities for various applications in visualization. Our work explores the utilization of the visual perception ability of multi-modal LLMs to develop Autonomous Visualization Agents (AVAs) that can interpret and accomplish user-defined visualization objectives through natural language. We propose the first framework for the design of AVAs and present several usage scenarios intended to demonstrate the general applicability of the proposed paradigm. The addition of visual perception allows AVAs to act as the virtual visualization assistant for domain experts who may lack the knowledge or expertise in fine-tuning visualization outputs. Our preliminary exploration and proof-of-concept agents suggest that this approach can be widely applicable whenever the choices of appropriate visualization parameters require the interpretation of previous visual output. Feedback from unstructured interviews with experts in AI research, medical visualization, and radiology has been incorporated, highlighting the practicality and potential of AVAs. Our study indicates that AVAs represent a general paradigm for designing intelligent visualization systems that can achieve high-level visualization goals, which pave the way for developing expert-level visualization agents in the future.