The information-theoretic value of unlabeled data in semi-supervised learning
This provides theoretical insights into when unlabeled data reduces labeled data requirements in semi-supervised learning, though it is incremental as it builds on existing learning theory frameworks.
The paper quantifies the benefit of unlabeled data in semi-supervised learning by proving a Θ(log n) multiplicative separation in labeled examples needed for learning projections over the Boolean hypercube with versus without knowledge of the unlabeled distribution, while showing no separation for all functions on any domain.
We quantify the separation between the numbers of labeled examples required to learn in two settings: Settings with and without the knowledge of the distribution of the unlabeled data. More specifically, we prove a separation by $Θ(\log n)$ multiplicative factor for the class of projections over the Boolean hypercube of dimension $n$. We prove that there is no separation for the class of all functions on domain of any size. Learning with the knowledge of the distribution (a.k.a. fixed-distribution learning) can be viewed as an idealized scenario of semi-supervised learning where the number of unlabeled data points is so great that the unlabeled distribution is known exactly. For this reason, we call the separation the value of unlabeled data.