Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning
This addresses the bottleneck of data annotation in text classification systems by improving stopping methods, but it is incremental as it focuses on forecasting techniques within existing active learning frameworks.
The study tackled the problem of forecasting text classification performance early in active learning to optimize annotation costs, finding that F-measure is harder to forecast than accuracy and that forecasting difficulty varies by algorithm, with decision trees being easiest and neural networks hardest.
When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stopping methods to minimize the cost of data annotation. An important capability for improving the utility of stopping methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and F-measure as the text classification system performance metrics and we find that F-measure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks.