Enhancing Image Quality Assessment Ability of LMMs via Retrieval-Augmented Generation
This provides a resource-efficient alternative for researchers and practitioners in computer vision, though it is incremental as it builds on existing training-free methods for LMMs.
The paper tackles the problem of improving Image Quality Assessment (IQA) in Large Multimodal Models (LMMs) without expensive fine-tuning, by introducing IQARAG, a training-free framework that uses Retrieval-Augmented Generation to retrieve reference images with quality scores, resulting in enhanced IQA performance across multiple datasets.
Large Multimodal Models (LMMs) have recently shown remarkable promise in low-level visual perception tasks, particularly in Image Quality Assessment (IQA), demonstrating strong zero-shot capability. However, achieving state-of-the-art performance often requires computationally expensive fine-tuning methods, which aim to align the distribution of quality-related token in output with image quality levels. Inspired by recent training-free works for LMM, we introduce IQARAG, a novel, training-free framework that enhances LMMs' IQA ability. IQARAG leverages Retrieval-Augmented Generation (RAG) to retrieve some semantically similar but quality-variant reference images with corresponding Mean Opinion Scores (MOSs) for input image. These retrieved images and input image are integrated into a specific prompt. Retrieved images provide the LMM with a visual perception anchor for IQA task. IQARAG contains three key phases: Retrieval Feature Extraction, Image Retrieval, and Integration & Quality Score Generation. Extensive experiments across multiple diverse IQA datasets, including KADID, KonIQ, LIVE Challenge, and SPAQ, demonstrate that the proposed IQARAG effectively boosts the IQA performance of LMMs, offering a resource-efficient alternative to fine-tuning for quality assessment.