Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
This work addresses interpretability and performance in multimodal AI for tasks like object detection, but it is incremental as it builds on existing vision-language models with a novel attention reweighting technique.
The paper tackles the problem of enhancing object localization in vision-language transformers for open-vocabulary referring object detection by proposing Reverse Contrast Attention (RCA), a plug-in method that improves FitAP scores by up to +26.6% in 11 out of 15 models without retraining.
We propose Reverse Contrast Attention (RCA), a plug-in method that enhances object localization in vision-language transformers without retraining. RCA reweights final-layer attention by suppressing extremes and amplifying mid-level activations to let semantically relevant but subdued tokens guide predictions. We evaluate it on Open Vocabulary Referring Object Detection (OV-RefOD), introducing FitAP, a confidence-free average precision metric based on IoU and box area. RCA improves FitAP in 11 out of 15 open-source VLMs, with gains up to $+26.6\%$. Effectiveness aligns with attention sharpness and fusion timing; while late-fusion models benefit consistently, models like $\texttt{DeepSeek-VL2}$ also improve, pointing to capacity and disentanglement as key factors. RCA offers both interpretability and performance gains for multimodal transformers. Codes and dataset are available from https://github.com/earl-juanico/rca