When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
For practitioners using VLMs in safety-critical applications, this work improves calibration reliability during test-time adaptation without sacrificing accuracy.
The paper addresses the problem of poor calibration in test-time adaptation of vision-language models (VLMs) due to entropy-driven overconfidence from prompt tuning. By modeling (class, attribute) pairs as nodes in a Symbolic Attribute Graph and using a Graph Attention Network with contrastive learning, the proposed method ARGTCA-DIV reduces average Expected Calibration Error (ECE) by ~37% over baselines across nine benchmarks.
Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.