EASE: An Easily-Customized Annotation System Powered by Efficiency Enhancement Mechanisms
This addresses the need for flexible and efficient annotation tools for NLP researchers, though it appears incremental as it builds on existing active learning and customization concepts.
The paper tackles the problem of annotation tools being difficult to customize and limited in active learning support by presenting EASE, a system with modular units and multiple back-end options, which significantly accelerates the annotation process for NLP researchers.
The performance of current supervised AI systems is tightly connected to the availability of annotated datasets. Annotations are usually collected through annotation tools, which are often designed for specific tasks and are difficult to customize. Moreover, existing annotation tools with an active learning mechanism often only support limited use cases. To address these limitations, we present EASE, an Easily-Customized Annotation System Powered by Efficiency Enhancement Mechanisms. \sysname provides modular annotation units for building customized annotation interfaces and also provides multiple back-end options that suggest annotations using (1) multi-task active learning; (2) demographic feature based active learning; (3) a prompt system that can query the API of large language models. We conduct multiple experiments and user studies to evaluate our system's flexibility and effectiveness. Our results show that our system can meet the diverse needs of NLP researchers and significantly accelerate the annotation process.