A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction
This addresses the challenge of limited and noisy datasets in sequential tasks like dialogue belief tracking, offering an incremental improvement in active learning methods.
The paper tackles the problem of reducing annotation costs and improving data quality in sequential tasks by introducing CAMEL, a self-supervised active learning framework that requires expert labeling for only part of a sequence and uses self-supervision for the rest, with results showing significant efficiency gains over baselines and improved dataset quality through label correction.
Supervised neural approaches are hindered by their dependence on large, meticulously annotated datasets, a requirement that is particularly cumbersome for sequential tasks. The quality of annotations tends to deteriorate with the transition from expert-based to crowd-sourced labelling. To address these challenges, we present CAMEL (Confidence-based Acquisition Model for Efficient self-supervised active Learning), a pool-based active learning framework tailored to sequential multi-output problems. CAMEL possesses two core features: (1) it requires expert annotators to label only a fraction of a chosen sequence, and (2) it facilitates self-supervision for the remainder of the sequence. By deploying a label correction mechanism, CAMEL can also be utilised for data cleaning. We evaluate CAMEL on two sequential tasks, with a special emphasis on dialogue belief tracking, a task plagued by the constraints of limited and noisy datasets. Our experiments demonstrate that CAMEL significantly outperforms the baselines in terms of efficiency. Furthermore, the data corrections suggested by our method contribute to an overall improvement in the quality of the resulting datasets.