Mitigating Transformer Overconfidence via Lipschitz Regularization
This addresses overconfidence issues in Transformers for computer vision applications, offering a novel regularization approach with theoretical guarantees.
The paper tackles the problem of Transformer overconfidence in computer vision by introducing Lipschitz Regularized Transformer (LRFormer), which improves prediction, calibration, and uncertainty estimation, outperforming state-of-the-art single forward pass methods on standard benchmarks.
Though Transformers have achieved promising results in many computer vision tasks, they tend to be over-confident in predictions, as the standard Dot Product Self-Attention (DPSA) can barely preserve distance for the unbounded input domain. In this work, we fill this gap by proposing a novel Lipschitz Regularized Transformer (LRFormer). Specifically, we present a new similarity function with the distance within Banach Space to ensure the Lipschitzness and also regularize the term by a contractive Lipschitz Bound. The proposed method is analyzed with a theoretical guarantee, providing a rigorous basis for its effectiveness and reliability. Extensive experiments conducted on standard vision benchmarks demonstrate that our method outperforms the state-of-the-art single forward pass approaches in prediction, calibration, and uncertainty estimation.