On the Informativeness of Supervision Signals
This work addresses the cost-benefit tradeoff in annotation strategies for supervised learning, providing a framework to optimize supervision costs for practitioners, though it is incremental in refining existing theoretical and empirical comparisons.
The paper tackles the problem of comparing the informativeness of different supervision signals (e.g., hard vs. soft labels) for representation learning, using information theory to show that hard labels are justified in big-data regimes while richer signals benefit few-shot learning and out-of-distribution generalization, validated with over 1 million image annotations.
Supervised learning typically focuses on learning transferable representations from training examples annotated by humans. While rich annotations (like soft labels) carry more information than sparse annotations (like hard labels), they are also more expensive to collect. For example, while hard labels only provide information about the closest class an object belongs to (e.g., "this is a dog"), soft labels provide information about the object's relationship with multiple classes (e.g., "this is most likely a dog, but it could also be a wolf or a coyote"). We use information theory to compare how a number of commonly-used supervision signals contribute to representation-learning performance, as well as how their capacity is affected by factors such as the number of labels, classes, dimensions, and noise. Our framework provides theoretical justification for using hard labels in the big-data regime, but richer supervision signals for few-shot learning and out-of-distribution generalization. We validate these results empirically in a series of experiments with over 1 million crowdsourced image annotations and conduct a cost-benefit analysis to establish a tradeoff curve that enables users to optimize the cost of supervising representation learning on their own datasets.