Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation
This addresses the cost-effectiveness of data labeling for practitioners in machine learning, though it is incremental as it builds on existing method-of-moments frameworks.
The paper tackles the problem of choosing between labeled and unlabeled data for latent variable estimation, showing that labeled points are worth a constant factor more than unlabeled points in well-specified models, but their relative value increases under misspecification due to additional bias, which can be reduced with a correction.
Labeling data for modern machine learning is expensive and time-consuming. Latent variable models can be used to infer labels from weaker, easier-to-acquire sources operating on unlabeled data. Such models can also be trained using labeled data, presenting a key question: should a user invest in few labeled or many unlabeled points? We answer this via a framework centered on model misspecification in method-of-moments latent variable estimation. Our core result is a bias-variance decomposition of the generalization error, which shows that the unlabeled-only approach incurs additional bias under misspecification. We then introduce a correction that provably removes this bias in certain cases. We apply our decomposition framework to three scenarios -- well-specified, misspecified, and corrected models -- to 1) choose between labeled and unlabeled data and 2) learn from their combination. We observe theoretically and with synthetic experiments that for well-specified models, labeled points are worth a constant factor more than unlabeled points. With misspecification, however, their relative value is higher due to the additional bias but can be reduced with correction. We also apply our approach to study real-world weak supervision techniques for dataset construction.