Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models
This addresses calibration issues for users of foundation models in low-shot classification, though it is an incremental method.
The paper tackles confidence calibration in foundation models for low-shot vision classification by proposing a confidence misalignment penalty (CMP) that redistributes logits during fine-tuning, resulting in an average improvement of 6.01% in Expected Calibration Error across multiple datasets.
Confidence calibration is an emerging challenge in real-world decision systems based on foundations models when used for downstream vision classification tasks. Due to various reasons exposed, logit scores on the CLIP head remain large irrespective of whether the image-language pairs reconcile. It is difficult to address in data space, given the few-shot regime. We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning, by moving an amount of log-likelihood to the true class commensurate to the relative amplitudes of the two likelihoods. We refer to it as \textit{confidence misalignment penalty (CMP)}. Extensive experiments on $12$ vision datasets and $5$ domain generalization datasets supports the calibration performance of our method against stat-of-the-art. CMP outperforms the benchmarked prompt learning methods, demonstrating average improvement in Expected Calibration Error (ECE) by average $6.01$\%, $4.01$ \% at minimum and $9.72$\% at maximum.