Tradeoffs in Resampling and Filtering for Imbalanced Classification
This work addresses imbalanced classification problems in natural language processing, which is incremental as it examines existing techniques rather than introducing new methods.
The paper investigates tradeoffs in model performance when selecting training data and filtering test data for imbalanced token classification tasks, finding that in highly imbalanced cases, filtering test data is as crucial as training data selection, with performance differences decreasing as the base rate of the rare class increases.
Imbalanced classification problems are extremely common in natural language processing and are solved using a variety of resampling and filtering techniques, which often involve making decisions on how to select training data or decide which test examples should be labeled by the model. We examine the tradeoffs in model performance involved in choices of training sample and filter training and test data in heavily imbalanced token classification task and examine the relationship between the magnitude of these tradeoffs and the base rate of the phenomenon of interest. In experiments on sequence tagging to detect rare phenomena in English and Arabic texts, we find that different methods of selecting training data bring tradeoffs in effectiveness and efficiency. We also see that in highly imbalanced cases, filtering test data using first-pass retrieval models is as important for model performance as selecting training data. The base rate of a rare positive class has a clear effect on the magnitude of the changes in performance caused by the selection of training or test data. As the base rate increases, the differences brought about by those choices decreases.