Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
It provides a comprehensive overview for researchers and practitioners working on enhancing AI systems with multimodal dynamic knowledge, but it is incremental as a survey rather than introducing new methods.
This survey tackles the problem of hallucinations and outdated knowledge in Large Language Models by analyzing Multimodal Retrieval-Augmented Generation systems, which integrate external dynamic information from multiple modalities like text, images, audio, and video to improve factual grounding.
Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey.