Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox
It addresses the problem of inadequate learning for tail categories in machine learning models across various domains, but it is incremental as it focuses on benchmarking and evaluation rather than proposing new methods.
This work tackles the challenge of long-tailed data distributions in domains like e-commerce and finance by developing HeroLT, a comprehensive benchmark that integrates 18 algorithms, 10 metrics, and 17 datasets across 6 tasks and 4 modalities, enabling effective evaluation of methods with 315 experiments.
Long-tailed data distributions pose challenges for a variety of domains like e-commerce, finance, biomedical science, and cyber security, where the performance of machine learning models is often dominated by head categories while tail categories are inadequately learned. This work aims to provide a systematic view of long-tailed learning with regard to three pivotal angles: (A1) the characterization of data long-tailedness, (A2) the data complexity of various domains, and (A3) the heterogeneity of emerging tasks. We develop HeroLT, a comprehensive long-tailed learning benchmark integrating 18 state-of-the-art algorithms, 10 evaluation metrics, and 17 real-world datasets across 6 tasks and 4 data modalities. HeroLT with novel angles and extensive experiments (315 in total) enables effective and fair evaluation of newly proposed methods compared with existing baselines on varying dataset types. Finally, we conclude by highlighting the significant applications of long-tailed learning and identifying several promising future directions. For accessibility and reproducibility, we open-source our benchmark HeroLT and corresponding results at https://github.com/SSSKJ/HeroLT.