Label Errors in the Tobacco3482 Dataset
This work exposes critical data quality problems in a widely used benchmark, which is incremental but important for researchers relying on accurate datasets.
The authors identified widespread label errors in the Tobacco3482 document classification dataset, finding that 11.7% of samples have improper annotations and 16.7% have multiple valid labels, and showed that 35% of mistakes by a top-performing model are due to these issues.
Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.