What is important about the No Free Lunch theorems?
This work provides foundational insights for machine learning and optimization by challenging assumptions about algorithm superiority and highlighting limitations in existing literature, with implications for philosophy of science.
The paper discusses the No Free Lunch theorems, which show that all induction algorithms perform equally under a uniform distribution over problems, and uses them to analyze non-uniform scenarios and compare algorithms without distributional assumptions, revealing that anti-cross-validation performs as well as cross-validation unless specific unformalized assumptions are made.
The No Free Lunch theorems prove that under a uniform distribution over induction problems (search problems or learning problems), all induction algorithms perform equally. As I discuss in this chapter, the importance of the theorems arises by using them to analyze scenarios involving {non-uniform} distributions, and to compare different algorithms, without any assumption about the distribution over problems at all. In particular, the theorems prove that {anti}-cross-validation (choosing among a set of candidate algorithms based on which has {worst} out-of-sample behavior) performs as well as cross-validation, unless one makes an assumption -- which has never been formalized -- about how the distribution over induction problems, on the one hand, is related to the set of algorithms one is choosing among using (anti-)cross validation, on the other. In addition, they establish strong caveats concerning the significance of the many results in the literature which establish the strength of a particular algorithm without assuming a particular distribution. They also motivate a ``dictionary'' between supervised learning and improve blackbox optimization, which allows one to ``translate'' techniques from supervised learning into the domain of blackbox optimization, thereby strengthening blackbox optimization algorithms. In addition to these topics, I also briefly discuss their implications for philosophy of science.