Stopping Active Learning based on Predicted Change of F Measure for Text Classification
This work addresses the data annotation bottleneck for users building text classification systems, but it is incremental as it focuses on improving stopping criteria within existing active learning frameworks.
The paper tackles the problem of determining when to stop active learning to limit annotation costs by introducing a new stopping method based on predicted change in F-measure, which estimates model performance changes at each iteration and can be applied with any base learner, resulting in a method that reduces the data annotation bottleneck in text classification systems.
During active learning, an effective stopping method allows users to limit the number of annotations, which is cost effective. In this paper, a new stopping method called Predicted Change of F Measure will be introduced that attempts to provide the users an estimate of how much performance of the model is changing at each iteration. This stopping method can be applied with any base learner. This method is useful for reducing the data annotation bottleneck encountered when building text classification systems.