A Scalable and Extensible Framework for Superposition-Structured Models
This work addresses the computational bottleneck in applying superposition-structured models to various datasets, offering a scalable solution for researchers and practitioners in machine learning and statistics.
The authors tackled the problem of efficiently solving superposition-structured statistical models, which combine multiple structural constraints for better interpretability and generalization, by developing a scalable and extensible proximal quasi-Newton framework that achieves super-linear convergence rates for models like fused sparse group lasso.
In many learning tasks, structural models usually lead to better interpretability and higher generalization performance. In recent years, however, the simple structural models such as lasso are frequently proved to be insufficient. Accordingly, there has been a lot of work on "superposition-structured" models where multiple structural constraints are imposed. To efficiently solve these "superposition-structured" statistical models, we develop a framework based on a proximal Newton-type method. Employing the smoothed conic dual approach with the LBFGS updating formula, we propose a scalable and extensible proximal quasi-Newton (SEP-QN) framework. Empirical analysis on various datasets shows that our framework is potentially powerful, and achieves super-linear convergence rate for optimizing some popular "superposition-structured" statistical models such as the fused sparse group lasso.