Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
This work addresses the challenge of enhancing REC accuracy for MLLMs in a training-free manner, which is incremental as it builds on existing techniques like Chain-of-Thought and tool use.
The paper tackles the problem of improving multimodal large language models (MLLMs) on referring expression comprehension (REC) without fine-tuning, by proposing a training-free framework called Chain-of-Caption that combines multiple visual and textual contexts, resulting in performance gains of 5% to 30% over baselines on accuracy at various IoU thresholds.
Given a textual description, the task of referring expression comprehension (REC) involves the localisation of the referred object in an image. Multimodal large language models (MLLMs) have achieved high accuracy on REC benchmarks through scaling up the model size and training data. Moreover, the performance of MLLMs can be further improved using techniques such as Chain-of-Thought and tool use, which provides additional visual or textual context to the model. In this paper, we analyse the effect of various techniques for providing additional visual and textual context via tool use to the MLLM and its effect on the REC task. Furthermore, we propose a training-free framework named Chain-of-Caption to improve the REC performance of MLLMs. We perform experiments on RefCOCO/RefCOCOg/RefCOCO+ and Ref-L4 datasets and show that individual textual or visual context can improve the REC performance without any fine-tuning. By combining multiple contexts, our training-free framework shows between 5% to 30% performance gain over the baseline model on accuracy at various Intersection over Union (IoU) thresholds.