Constrained Variable Projection for Structured Problems
For practitioners solving separable nonlinear least-squares problems with convex constraints, this provides a principled and efficient optimization framework.
The paper extends variable projection to constrained bilevel optimization for structured data-science models, deriving exact reduced gradients and a conditional-gradient algorithm. Experiments on sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning show improved wall-clock and data efficiency over joint optimization.
Variable projection is a classical technique for separable nonlinear least-squares problems, in which variables that enter linearly are eliminated exactly, yielding a reduced nonlinear problem. By expressing this framework as a particular instance of a broader class of bilevel optimization problems, we develop a constrained variable-projection framework for data-science models, where the remaining variables are subject to convex constraints and the eliminated variables arise from a lower-level least-squares problem. In particular, by interpreting variable projection as a collapsed bilevel optimization problem, we derive exact reduced-gradient formulas compatible with automatic differentiation and propose a conditional-gradient algorithm for the resulting constrained reduced problem. We establish convergence guarantees under standard smoothness and compactness assumptions, and discuss extensions to structured lower-level variables. Numerical experiments on sparse autoencoding, dictionary learning, blind deconvolution, and few-shot learning suggest that the method can improve wall-clock efficiency and data efficiency relative to natural joint-optimization baselines.