Interactive Machine Teaching by Labeling Rules and Instances
This work addresses the problem of reducing labeling costs for experts in weakly supervised learning, offering a practical solution that is incremental but with strong performance gains.
The paper tackles the challenge of efficiently using expert time for data supervision by comparing rule creation and instance labeling, and introduces INTERVAL, an interactive framework that combines automated rule extraction with expert feedback. INTERVAL outperforms state-of-the-art weakly supervised methods by 7% in F1 and requires only 10 expert queries to surpass the performance of active learning methods with 100 queries.
Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert's limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.