Describe-to-Score: Text-Guided Efficient Image Complexity Assessment
This work addresses the limitation of existing methods that rely solely on visual features, offering a more efficient and accurate solution for computer vision applications, though it is incremental in its use of multi-modal fusion.
The paper tackles the problem of image complexity assessment by introducing a vision-text fusion approach that integrates visual and textual semantic features, resulting in improved accuracy and generalization on the IC9600 dataset and competitive performance on no-reference image quality assessment benchmarks.
Accurately assessing image complexity (IC) is critical for computer vision, yet most existing methods rely solely on visual features and often neglect high-level semantic information, limiting their accuracy and generalization. We introduce vision-text fusion for IC modeling. This approach integrates visual and textual semantic features, increasing representational diversity. It also reduces the complexity of the hypothesis space, which enhances both accuracy and generalization in complexity assessment. We propose the D2S (Describe-to-Score) framework, which generates image captions with a pre-trained vision-language model. We propose the feature alignment and entropy distribution alignment mechanisms, D2S guides semantic information to inform complexity assessment while bridging the gap between vision and text modalities. D2S utilizes multi-modal information during training but requires only the vision branch during inference, thereby avoiding multi-modal computational overhead and enabling efficient assessment. Experimental results demonstrate that D2S outperforms existing methods on the IC9600 dataset and maintains competitiveness on no-reference image quality assessment (NR-IQA) benchmark, validating the effectiveness and efficiency of multi-modal fusion in complexity-related tasks. Code is available at: https://github.com/xauat-liushipeng/D2S