Instance-wise Supervision-level Optimization in Active Learning
This work addresses label efficiency in machine learning for researchers and practitioners, offering an incremental improvement by integrating weak supervision into active learning.
The paper tackles the problem of optimizing annotation levels in active learning by introducing the ISO framework, which selects instances and determines their optimal annotation level within a fixed budget, resulting in higher accuracy at lower cost compared to traditional and state-of-the-art methods.
Active learning (AL) is a label-efficient machine learning paradigm that focuses on selectively annotating high-value instances to maximize learning efficiency. Its effectiveness can be further enhanced by incorporating weak supervision, which uses rough yet cost-effective annotations instead of exact (i.e., full) but expensive annotations. We introduce a novel AL framework, Instance-wise Supervision-Level Optimization (ISO), which not only selects the instances to annotate but also determines their optimal annotation level within a fixed annotation budget. Its optimization criterion leverages the value-to-cost ratio (VCR) of each instance while ensuring diversity among the selected instances. In classification experiments, ISO consistently outperforms traditional AL methods and surpasses a state-of-the-art AL approach that combines full and weak supervision, achieving higher accuracy at a lower overall cost. This code is available at https://github.com/matsuo-shinnosuke/ISOAL.