The Interplay Between Interpolation and Aggregation in Regression: Optimal Sample Complexity
For learning theorists, it provides a precise characterization of learnability in regression with interpolation and aggregation, revealing fundamental limitations of finite interpolating aggregation.
This paper characterizes the sample complexity of regression with interpolation and aggregation, showing that a simple median-based aggregation of three interpolating hypotheses is optimal and strictly more powerful than proper learning. It also identifies hypothesis classes that require infinitely many hypotheses or non-interpolating rules for learnability.
This work investigates theoretically the interplay between interpolation and aggregation in regression. We establish that the $γ$-graph dimension characterizes learnability for a broad class of natural aggregation procedures. Furthermore, we prove that an extremely simple aggregation procedure, combining three interpolating hypotheses via the median, is optimal among all these aggregation procedures, and is strictly more powerful than proper learning. Finally, we show that some hypothesis classes are learnable only by aggregating infinitely many hypotheses or by using non-interpolating aggregation rules (which may predict outside the range of their inputs), and any finite interpolating aggregation fails to achieve even trivial performance.