Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement
This addresses reliability issues for practical deep learning deployments, but it is incremental as it builds on existing methods for confidence calibration.
The paper tackles the problem of model unreliability due to overconfidence by estimating reliability through inter-model latent agreement, showing that fusing this agreement into predictive confidence significantly improves reliability across in-distribution and out-of-distribution settings.
Reliable application of machine learning is of primary importance to the practical deployment of deep learning methods. A fundamental challenge is that models are often unreliable due to overconfidence. In this paper, we estimate a model's reliability by measuring \emph{the agreement between its latent space, and the latent space of a foundation model}. However, it is challenging to measure the agreement between two different latent spaces due to their incoherence, \eg, arbitrary rotations and different dimensionality. To overcome this incoherence issue, we design a \emph{neighborhood agreement measure} between latent spaces and find that this agreement is surprisingly well-correlated with the reliability of a model's predictions. Further, we show that fusing neighborhood agreement into a model's predictive confidence in a post-hoc way significantly improves its reliability. Theoretical analysis and extensive experiments on failure detection across various datasets verify the effectiveness of our method on both in-distribution and out-of-distribution settings.