Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts
This addresses the issue of inefficient and inaccurate fine-tuning in long-tail learning for machine learning practitioners, offering an incremental improvement with a new algorithm.
The paper tackles the problem of performance deterioration in long-tail learning when using heavy fine-tuning with foundation models, finding that lightweight fine-tuning is more effective and leads to reduced training time and parameters with improved accuracy compared to state-of-the-art methods.
The fine-tuning paradigm in addressing long-tail learning tasks has sparked significant interest since the emergence of foundation models. Nonetheless, how fine-tuning impacts performance in long-tail learning was not explicitly quantified. In this paper, we disclose that heavy fine-tuning may even lead to non-negligible performance deterioration on tail classes, and lightweight fine-tuning is more effective. The reason is attributed to inconsistent class conditions caused by heavy fine-tuning. With the observation above, we develop a low-complexity and accurate long-tail learning algorithms LIFT with the goal of facilitating fast prediction and compact models by adaptive lightweight fine-tuning. Experiments clearly verify that both the training time and the learned parameters are significantly reduced with more accurate predictive performance compared with state-of-the-art approaches. The implementation code is available at https://github.com/shijxcs/LIFT.