Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark
This work addresses the problem of assessing multimodal LLMs' capabilities in visual network analysis for researchers and practitioners, but it is incremental as it introduces a new benchmark without proposing a solution.
The paper evaluated GPT-4 and LLaVa on zero-shot Visual Network Analysis tasks, finding that both models struggled with basic graph analysis, with GPT-4 performing better but still failing on tasks like identifying maximal degree nodes and counting components.
We evaluate the zero-shot ability of GPT-4 and LLaVa to perform simple Visual Network Analysis (VNA) tasks on small-scale graphs. We evaluate the Vision Language Models (VLMs) on 5 tasks related to three foundational network science concepts: identifying nodes of maximal degree on a rendered graph, identifying whether signed triads are balanced or unbalanced, and counting components. The tasks are structured to be easy for a human who understands the underlying graph theoretic concepts, and can all be solved by counting the appropriate elements in graphs. We find that while GPT-4 consistently outperforms LLaVa, both models struggle with every visual network analysis task we propose. We publicly release the first benchmark for the evaluation of VLMs on foundational VNA tasks.