Convex Relaxation Regression: Black-Box Optimization of Smooth Functions by Learning Their Convex Envelopes
This provides a general-purpose strategy for estimating convex relaxations, enabling convex optimization tools to solve a class of non-convex problems, which is a significant advance for machine learning and optimization theory.
The paper tackles the challenge of solving non-convex optimization problems by introducing Convex Relaxation Regression (CoRR), a method that learns convex relaxations for smooth functions by fitting convex functions to random evaluations, with proven convergence to the global optimizer with error O((log(1/δ)/T)^α).
Finding efficient and provable methods to solve non-convex optimization problems is an outstanding challenge in machine learning and optimization theory. A popular approach used to tackle non-convex problems is to use convex relaxation techniques to find a convex surrogate for the problem. Unfortunately, convex relaxations typically must be found on a problem-by-problem basis. Thus, providing a general-purpose strategy to estimate a convex relaxation would have a wide reaching impact. Here, we introduce Convex Relaxation Regression (CoRR), an approach for learning convex relaxations for a class of smooth functions. The main idea behind our approach is to estimate the convex envelope of a function $f$ by evaluating $f$ at a set of $T$ random points and then fitting a convex function to these function evaluations. We prove that with probability greater than $1-δ$, the solution of our algorithm converges to the global optimizer of $f$ with error $\mathcal{O} \Big( \big(\frac{\log(1/δ) }{T} \big)^α \Big)$ for some $α> 0$. Our approach enables the use of convex optimization tools to solve a class of non-convex optimization problems.