Non-smooth Variable Projection
It extends variable projection to non-smooth cases, which is incremental but useful for practitioners in machine learning and inverse problems.
The paper tackles optimization problems with non-smooth terms and inexact projections by proposing an inexact adaptive algorithm, analyzing its computational complexity, and applying it to machine learning and inverse problems.
Variable projection solves structured optimization problems by completely minimizing over a subset of the variables while iterating over the remaining variables. Over the last 30 years, the technique has been widely used, with empirical and theoretical results demonstrating both greater efficacy and greater stability compared to competing approaches. Classic examples have exploited closed-form projections and smoothness of the objective function. We extend the approach to problems that include non-smooth terms, and where the projection subproblems can only be solved inexactly by iterative methods. We propose an inexact adaptive algonrithm for solving such problems and analyze its computational complexity. Finally, we show how the theory can be used to design methods for selected problems occurring frequently in machine-learning and inverse problems.