No Free Lunch Theorem and Bayesian probability theory: two sides of the same coin. Some implications for black-box optimization and metaheuristics
This work addresses theoretical issues with implications for practitioners in industrial design and optimization, though it appears incremental in linking existing concepts.
The paper tackles the challenge of solving hard optimization problems with many variables and complex fitness landscapes by interpreting the No Free Lunch theorem to derive a Bayesian optimization framework, emphasizing that selecting a prior over functions is essential for black-box optimization.
Challenging optimization problems, which elude acceptable solution via conventional calculus methods, arise commonly in different areas of industrial design and practice. Hard optimization problems are those who manifest the following behavior: a) high number of independent input variables; b) very complex or irregular multi-modal fitness; c) computational expensive fitness evaluation. This paper will focus on some theoretical issues that have strong implications for practice. I will stress how an interpretation of the No Free Lunch theorem leads naturally to a general Bayesian optimization framework. The choice of a prior over the space of functions is a critical and inevitable step in every black-box optimization.