PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models
This addresses a domain-specific problem in vision-language models for image classification, offering an incremental improvement over existing methods.
The paper tackles the problem of deterministic embeddings in language-guided attention frameworks overlooking multivaluedness in cross-modal mappings, introducing PARIC to generate probabilistic reference attention maps that improve alignment and incorporate uncertainty. Results show PARIC enhances prediction accuracy, mitigates bias, ensures consistency, and improves robustness across datasets.
Language-guided attention frameworks have significantly enhanced both interpretability and performance in image classification; however, the reliance on deterministic embeddings from pre-trained vision-language foundation models to generate reference attention maps frequently overlooks the intrinsic multivaluedness and ill-posed characteristics of cross-modal mappings. To address these limitations, we introduce PARIC, a probabilistic framework for guiding visual attention via language specifications. Our approach enables pre-trained vision-language models to generate probabilistic reference attention maps, which align textual and visual modalities more effectively while incorporating uncertainty estimates, as compared to their deterministic counterparts. Experiments on benchmark test problems demonstrate that PARIC enhances prediction accuracy, mitigates bias, ensures consistent predictions, and improves robustness across various datasets.