Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering
This work addresses challenges in VQA for AI systems by enabling autonomous tool use and team collaboration, though it is incremental as it builds on existing LLM methods.
The paper tackles the problem of knowledge-based Visual Question Answering (VQA) by proposing a multi-agent voting framework using Large Language Models (LLMs) to simulate human-like tool usage and collaboration, achieving improvements of 2.2 and 1.0 points over baselines on OK-VQA and A-OKVQA datasets.
Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.