Hedging on the Frontier: Learning New Tasks with Few Samples
This work provides a method for improving learning performance on new tasks with few samples, which is relevant for practitioners and researchers dealing with data scarcity.
This paper addresses the challenge of learning new tasks with limited samples by leveraging side information from public benchmarks. The authors observe that weak monotonicity often holds, meaning models performing well on many benchmarks tend to do well on new tasks. They explore this property within transfer learning and model selection aggregation, showing it allows for pruning model classes and adapting to trade-off geometries.
When a learner faces a new task with few samples, it must leverage any available side information. In practice, this often comes in the form of model evaluations on related tasks in public benchmarks. A key question then is how to model task relatedness such that it is both realistic and the benchmark evaluations lead to provable gains. Empirically, we observe that weak monotonicity is often approximately satisfied: if a model dominates another on many benchmarks, it also tends to outperform on the new task. We explore the statistical complexity of learning under (approximate) weak monotonicity, leveraging it within two learning paradigms: transfer learning and model selection aggregation. We show that not only can we prune the model class based on monotonicity, but we can also further adapt to the geometry of the available trade-offs by hedging on the frontier.