Towards Realistic Single-Task Continuous Learning Research for NER
It provides a domain-specific tool for NLP researchers to advance continuous learning, but is incremental as it adapts existing data rather than introducing new methods.
The paper addresses the lack of realistic benchmarks for continuous learning in NLP by constructing a CL NER dataset from an existing public dataset, studying challenges like accuracy loss and the effectiveness of data rehearsal.
There is an increasing interest in continuous learning (CL), as data privacy is becoming a priority for real-world machine learning applications. Meanwhile, there is still a lack of academic NLP benchmarks that are applicable for realistic CL settings, which is a major challenge for the advancement of the field. In this paper we discuss some of the unrealistic data characteristics of public datasets, study the challenges of realistic single-task continuous learning as well as the effectiveness of data rehearsal as a way to mitigate accuracy loss. We construct a CL NER dataset from an existing publicly available dataset and release it along with the code to the research community.