GradTail: Learning Long-Tailed Data Using Gradient-based Sample Weighting
This work addresses the challenge of handling long-tailed data in machine learning, which is a common issue in real-world datasets, but it is incremental as it builds on existing gradient-based methods.
The paper tackles the problem of improving model performance on long-tailed data distributions by proposing GradTail, an algorithm that uses gradient dot product agreement to dynamically upweight long-tailed samples during training, leading to performance improvements for both classification and regression models.
We propose GradTail, an algorithm that uses gradients to improve model performance on the fly in the face of long-tailed training data distributions. Unlike conventional long-tail classifiers which operate on converged - and possibly overfit - models, we demonstrate that an approach based on gradient dot product agreement can isolate long-tailed data early on during model training and improve performance by dynamically picking higher sample weights for that data. We show that such upweighting leads to model improvements for both classification and regression models, the latter of which are relatively unexplored in the long-tail literature, and that the long-tail examples found by gradient alignment are consistent with our semantic expectations.