ConjNorm: Tractable Density Estimation for Out-of-Distribution Detection
This work addresses the challenge of reliable OOD detection in machine learning, offering a novel approach that significantly outperforms existing methods, though it is incremental in advancing density-based score design.
The paper tackled the problem of out-of-distribution (OOD) detection by proposing a theoretical framework based on Bregman divergence and the ConjNorm method, which reframes density function design as optimizing a norm coefficient, resulting in state-of-the-art performance with improvements of up to 13.25% and 28.19% in FPR95 on benchmarks like CIFAR-100 and ImageNet-1K.
Post-hoc out-of-distribution (OOD) detection has garnered intensive attention in reliable machine learning. Many efforts have been dedicated to deriving score functions based on logits, distances, or rigorous data distribution assumptions to identify low-scoring OOD samples. Nevertheless, these estimate scores may fail to accurately reflect the true data density or impose impractical constraints. To provide a unified perspective on density-based score design, we propose a novel theoretical framework grounded in Bregman divergence, which extends distribution considerations to encompass an exponential family of distributions. Leveraging the conjugation constraint revealed in our theorem, we introduce a \textsc{ConjNorm} method, reframing density function design as a search for the optimal norm coefficient $p$ against the given dataset. In light of the computational challenges of normalization, we devise an unbiased and analytically tractable estimator of the partition function using the Monte Carlo-based importance sampling technique. Extensive experiments across OOD detection benchmarks empirically demonstrate that our proposed \textsc{ConjNorm} has established a new state-of-the-art in a variety of OOD detection setups, outperforming the current best method by up to 13.25$\%$ and 28.19$\%$ (FPR95) on CIFAR-100 and ImageNet-1K, respectively.