FAIR: Filtering of Automatically Induced Rules
This addresses the bottleneck of requiring diverse, high-quality rules in weak supervision for domain-specific applications, offering an incremental improvement in rule-filtering methods.
The paper tackles the problem of filtering high-quality rules from automatically induced rules in weak supervision, proposing an algorithm that uses submodular objective functions to account for precision, coverage, and conflicts, and shows superior performance on text classification datasets with statistically significant results compared to existing approaches.
The availability of large annotated data can be a critical bottleneck in training machine learning algorithms successfully, especially when applied to diverse domains. Weak supervision offers a promising alternative by accelerating the creation of labeled training data using domain-specific rules. However, it requires users to write a diverse set of high-quality rules to assign labels to the unlabeled data. Automatic Rule Induction (ARI) approaches circumvent this problem by automatically creating rules from features on a small labeled set and filtering a final set of rules from them. In the ARI approach, the crucial step is to filter out a set of a high-quality useful subset of rules from the large set of automatically created rules. In this paper, we propose an algorithm (Filtering of Automatically Induced Rules) to filter rules from a large number of automatically induced rules using submodular objective functions that account for the collective precision, coverage, and conflicts of the rule set. We experiment with three ARI approaches and five text classification datasets to validate the superior performance of our algorithm with respect to several semi-supervised label aggregation approaches. Further, we show that achieves statistically significant results in comparison to existing rule-filtering approaches.