Beyond Logit Adjustment: A Residual Decomposition Framework for Long-Tailed Reranking
This work addresses long-tailed classification challenges for domains like image recognition and medical diagnosis, offering a novel decomposition that clarifies when pairwise correction is beneficial, though it is incremental in advancing existing post-hoc methods.
The paper tackles the problem of long-tailed classification by proposing a residual decomposition framework for reranking, showing that a fixed classwise offset is insufficient when ordering constraints vary across inputs, and introduces REPAIR, a lightweight post-hoc reranker that improves performance on five benchmarks.
Long-tailed classification, where a small number of frequent classes dominate many rare ones, remains challenging because models systematically favor frequent classes at inference time. Existing post-hoc methods such as logit adjustment address this by adding a fixed classwise offset to the base-model logits. However, the correction required to restore the relative ranking of two classes need not be constant across inputs, and a fixed offset cannot adapt to such variation. We study this problem through Bayes-optimal reranking on a base-model top-k shortlist. The gap between the optimal score and the base score, the residual correction, decomposes into a classwise component that is constant within each class, and a pairwise component that depends on the input and competing labels. When the residual is purely classwise, a fixed offset suffices to recover the Bayes-optimal ordering. We further show that when the same label pair induces incompatible ordering constraints across contexts, no fixed offset can achieve this recovery. This decomposition leads to testable predictions regarding when pairwise correction can improve performance and when cannot. We develop REPAIR (Reranking via Pairwise residual correction), a lightweight post-hoc reranker that combines a shrinkage-stabilized classwise term with a linear pairwise term driven by competition features on the shortlist. Experiments on five benchmarks spanning image classification, species recognition, scene recognition, and rare disease diagnosis confirm that the decomposition explains where pairwise correction helps and where classwise correction alone suffices.