Data Programming by Demonstration: A Framework for Interactively Learning Labeling Functions
This addresses the problem of reducing the burden of creating labeling functions for domain experts without programming skills, though it is incremental as it builds on existing data programming methods.
The paper tackles the difficulty of writing labeling functions for data programming by introducing a framework called data programming by demonstration (DPBD), which uses interactive user demonstrations to generate rules, and finds in a user study with 10 data scientists that it is easier to use and learn while maintaining comparable model performance.
Data programming is a programmatic weak supervision approach to efficiently curate large-scale labeled training data. Writing data programs (labeling functions) requires, however, both programming literacy and domain expertise. Many subject matter experts have neither programming proficiency nor time to effectively write data programs. Furthermore, regardless of one's expertise in coding or machine learning, transferring domain expertise into labeling functions by enumerating rules and thresholds is not only time consuming but also inherently difficult. Here we propose a new framework, data programming by demonstration (DPBD), to generate labeling rules using interactive demonstrations of users. DPBD aims to relieve the burden of writing labeling functions from users, enabling them to focus on higher-level semantics such as identifying relevant signals for labeling tasks. We operationalize our framework with Ruler, an interactive system that synthesizes labeling rules for document classification by using span-level annotations of users on document examples. We compare Ruler with conventional data programming through a user study conducted with 10 data scientists creating labeling functions for sentiment and spam classification tasks. We find that Ruler is easier to use and learn and offers higher overall satisfaction, while providing discriminative model performances comparable to ones achieved by conventional data programming.