Weak Supervision with Incremental Source Accuracy Estimation
This work addresses the need for efficient label generation in dynamic environments, though it is incremental as it builds on existing offline methods.
The paper tackles the problem of generating labels for real-time data by developing a method to incrementally estimate the dependency structure and accuracy of weak supervision sources, resulting in probabilistic labels with accuracy matching existing offline methods.
Motivated by the desire to generate labels for real-time data we develop a method to estimate the dependency structure and accuracy of weak supervision sources incrementally. Our method first estimates the dependency structure associated with the supervision sources and then uses this to iteratively update the estimated source accuracies as new data is received. Using both off-the-shelf classification models trained using publicly-available datasets and heuristic functions as supervision sources we show that our method generates probabilistic labels with an accuracy matching that of existing off-line methods.