Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect
This addresses the problem of biased predictions in long-tailed datasets for computer vision tasks, offering a novel theoretical approach rather than incremental improvements.
The paper tackles long-tailed classification by identifying SGD momentum as a confounder with harmful effects on tail predictions and beneficial effects on representation learning, proposing a causal inference framework to remove the bad and keep the good effects. It achieves new state-of-the-art results on benchmarks like Long-tailed CIFAR-10/-100, ImageNet-LT, and LVIS.
As the class size grows, maintaining a balanced dataset across many classes is challenging because the data are long-tailed in nature; it is even impossible when the sample-of-interest co-exists with each other in one collectable unit, e.g., multiple visual instances in one image. Therefore, long-tailed classification is the key to deep learning at scale. However, existing methods are mainly based on re-weighting/re-sampling heuristics that lack a fundamental theory. In this paper, we establish a causal inference framework, which not only unravels the whys of previous methods, but also derives a new principled solution. Specifically, our theory shows that the SGD momentum is essentially a confounder in long-tailed classification. On one hand, it has a harmful causal effect that misleads the tail prediction biased towards the head. On the other hand, its induced mediation also benefits the representation learning and head prediction. Our framework elegantly disentangles the paradoxical effects of the momentum, by pursuing the direct causal effect caused by an input sample. In particular, we use causal intervention in training, and counterfactual reasoning in inference, to remove the "bad" while keep the "good". We achieve new state-of-the-arts on three long-tailed visual recognition benchmarks: Long-tailed CIFAR-10/-100, ImageNet-LT for image classification and LVIS for instance segmentation.