NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning
This package lowers the technical barrier for machine learning practitioners by providing a scalable tool for nonconvex optimization, though it is incremental as it ports an existing solver to a more accessible framework.
The authors tackled the difficulty of solving nonconvex optimization problems in machine learning, especially for nonsmooth and constrained cases, by introducing NCVX, a user-friendly package that enables solving general constrained deep learning problems with features like auto-differentiation and GPU acceleration.
Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind. NCVX is available at https://ncvx.org, with detailed documentation and numerous examples from machine learning and other fields.