Feed-Forward Neural Networks as a Mixed-Integer Program
This work addresses training challenges in neural networks for researchers in optimization and machine learning, but it appears incremental as it builds on existing MIP formulations for fixed-parameter DNNs.
The study tackled the problem of training neural networks by formulating trained ReLU neurons as a mixed-integer program (MIP) and applied MIP models to train various architectures, including binary and binarized DNNs, with experiments on handwritten digit classification showing comparative performance assessments.
Deep neural networks (DNNs) are widely studied in various applications. A DNN consists of layers of neurons that compute affine combinations, apply nonlinear operations, and produce corresponding activations. The rectified linear unit (ReLU) is a typical nonlinear operator, outputting the max of its input and zero. In scenarios like max pooling, where multiple input values are involved, a fixed-parameter DNN can be modeled as a mixed-integer program (MIP). This formulation, with continuous variables representing unit outputs and binary variables for ReLU activation, finds applications across diverse domains. This study explores the formulation of trained ReLU neurons as MIP and applies MIP models for training neural networks (NNs). Specifically, it investigates interactions between MIP techniques and various NN architectures, including binary DNNs (employing step activation functions) and binarized DNNs (with weights and activations limited to $-1,0,+1$). The research focuses on training and evaluating proposed approaches through experiments on handwritten digit classification models. The comparative study assesses the performance of trained ReLU NNs, shedding light on the effectiveness of MIP formulations in enhancing training processes for NNs.