Stratified Sampling for Extreme Multi-Label Data
This addresses the issue of poor generalization and unreliable performance estimates in XML research, which is incremental as it adapts stratification methods from binary/multi-class settings to XML.
The paper tackled the problem of generating stratified partitions for extreme multi-label classification datasets, which was previously unavailable, and presented a new algorithm that efficiently creates such partitions for datasets with millions of labels.
Extreme multi-label classification (XML) is becoming increasingly relevant in the era of big data. Yet, there is no method for effectively generating stratified partitions of XML datasets. Instead, researchers typically rely on provided test-train splits that, 1) aren't always representative of the entire dataset, and 2) are missing many of the labels. This can lead to poor generalization ability and unreliable performance estimates, as has been established in the binary and multi-class settings. As such, this paper presents a new and simple algorithm that can efficiently generate stratified partitions of XML datasets with millions of unique labels. We also examine the label distributions of prevailing benchmark splits, and investigate the issues that arise from using unrepresentative subsets of data for model development. The results highlight the difficulty of stratifying XML data, and demonstrate the importance of using stratified partitions for training and evaluation.