Efficient Meta Subspace Optimization
This work addresses efficiency limitations in large-scale optimization for computational applications, though it appears incremental as it builds on existing subspace optimization frameworks.
The paper tackles the suboptimality of fixed subspace update policies in large-scale optimization by proposing a Meta Subspace Optimization (MSO) framework that dynamically determines the subspace matrix at each iteration, resulting in significant performance improvements over existing methods.
Subspace optimization methods have the attractive property of reducing large-scale optimization problems to a sequence of low-dimensional subspace optimization problems. However, existing subspace optimization frameworks adopt a fixed update policy of the subspace and therefore appear to be sub-optimal. In this paper, we propose a new \emph{Meta Subspace Optimization} (MSO) framework for large-scale optimization problems, which allows to determine the subspace matrix at each optimization iteration. In order to remain invariant to the optimization problem's dimension, we design an \emph{efficient} meta optimizer based on very low-dimensional subspace optimization coefficients, inducing a rule-based method that can significantly improve performance. Finally, we design and analyze a reinforcement learning (RL) procedure based on the subspace optimization dynamics whose learnt policies outperform existing subspace optimization methods.