Long-tail learning via logit adjustment
This addresses the challenge of generalization on rare labels in real-world classification problems, which is incremental as it builds on existing logit adjustment ideas.
The paper tackles the problem of imbalanced or long-tailed label distributions in classification by proposing two simple modifications to softmax cross-entropy training based on logit adjustment, which unify and generalize recent methods with improved statistical grounding and empirical performance.
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.