Explainable Multi-Modal Data Exploration in Natural Language via LLM Agent
This addresses a widely unexplored challenge for enterprises, organizations, or hospitals needing to explore multi-modal data systems, representing a novel method rather than an incremental improvement.
The paper tackles the problem of querying multi-modal data (e.g., databases and images) in natural language by proposing XMODE, a system that uses an LLM-based agent to decompose questions into subtasks like text-to-SQL and image analysis, achieving state-of-the-art performance in accuracy, query latency, API costs, planning efficiency, and explanation quality.
International enterprises, organizations, or hospitals collect large amounts of multi-modal data stored in databases, text documents, images, and videos. While there has been recent progress in the separate fields of multi-modal data exploration as well as in database systems that automatically translate natural language questions to database query languages, the research challenge of querying database systems combined with other unstructured modalities such as images in natural language is widely unexplored. In this paper, we propose XMODE - a system that enables explainable, multi-modal data exploration in natural language. Our approach is based on the following research contributions: (1) Our system is inspired by a real-world use case that enables users to explore multi-modal information systems. (2) XMODE leverages a LLM-based agentic AI framework to decompose a natural language question into subtasks such as text-to-SQL generation and image analysis. (3) Experimental results on multi-modal datasets over relational data and images demonstrate that our system outperforms state-of-the-art multi-modal exploration systems, excelling not only in accuracy but also in various performance metrics such as query latency, API costs, planning efficiency, and explanation quality, thanks to the more effective utilization of the reasoning capabilities of LLMs.