Fast Global Convergence via Landscape of Empirical Loss
This work addresses the problem of efficient optimization for non-convex loss functions in machine learning, which is incremental as it builds on prior landscape analysis.
The paper tackles the computational challenge of optimizing non-convex M-estimators by showing that stochastic variance reduction methods achieve global optimality with linear convergence rates, leveraging statistical properties of the population loss.
While optimizing convex objective (loss) functions has been a powerhouse for machine learning for at least two decades, non-convex loss functions have attracted fast growing interests recently, due to many desirable properties such as superior robustness and classification accuracy, compared with their convex counterparts. The main obstacle for non-convex estimators is that it is in general intractable to find the optimal solution. In this paper, we study the computational issues for some non-convex M-estimators. In particular, we show that the stochastic variance reduction methods converge to the global optimal with linear rate, by exploiting the statistical property of the population loss. En route, we improve the convergence analysis for the batch gradient method in \cite{mei2016landscape}.