Nonlinear Optimization with GPU-Accelerated Neural Network Constraints
This work addresses computational efficiency in optimization problems with neural network constraints, which is incremental for researchers and practitioners in machine learning and engineering domains.
The authors tackled the problem of optimizing over trained neural networks by proposing a reduced-space formulation that treats networks as gray boxes, leading to faster solves and fewer iterations in interior point methods, with demonstrations on adversarial generation for MNIST and security-constrained optimal power flow.
We propose a reduced-space formulation for optimizing over trained neural networks where the network's outputs and derivatives are evaluated on a GPU. To do this, we treat the neural network as a "gray box" where intermediate variables and constraints are not exposed to the optimization solver. Compared to the full-space formulation, in which intermediate variables and constraints are exposed to the optimization solver, the reduced-space formulation leads to faster solves and fewer iterations in an interior point method. We demonstrate the benefits of this method on two optimization problems: Adversarial generation for a classifier trained on MNIST images and security-constrained optimal power flow with transient feasibility enforced using a neural network surrogate.