High-Dimensional Optimization in Adaptive Random Subspaces
This work addresses optimization efficiency for high-dimensional problems in machine learning, offering incremental improvements over existing randomized methods.
The paper tackles high-dimensional optimization by proposing an adaptive random subspace method that generalizes coordinate descent, showing it significantly outperforms oblivious sampling with improvements characterized by data matrix spectrum, and demonstrates speed ups in machine learning tasks like logistic regression and neural networks.
We propose a new randomized optimization method for high-dimensional problems which can be seen as a generalization of coordinate descent to random subspaces. We show that an adaptive sampling strategy for the random subspace significantly outperforms the oblivious sampling method, which is the common choice in the recent literature. The adaptive subspace can be efficiently generated by a correlated random matrix ensemble whose statistics mimic the input data. We prove that the improvement in the relative error of the solution can be tightly characterized in terms of the spectrum of the data matrix, and provide probabilistic upper-bounds. We then illustrate the consequences of our theory with data matrices of different spectral decay. Extensive experimental results show that the proposed approach offers significant speed ups in machine learning problems including logistic regression, kernel classification with random convolution layers and shallow neural networks with rectified linear units. Our analysis is based on convex analysis and Fenchel duality, and establishes connections to sketching and randomized matrix decomposition.