An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions
This work addresses the challenge of optimizing binary activation neural networks for specific datasets, but it is incremental as it builds on existing integer programming methods.
The authors tackled the problem of training deep neural networks with binary activation functions by reformulating them as mixed-integer linear programs solvable to global optimality, and showed that a heuristic version outperformed classical deep neural networks on the Breast Cancer Wisconsin dataset but performed worse on random data.
We study deep neural networks with binary activation functions (BDNN), i.e. the activation function only has two states. We show that the BDNN can be reformulated as a mixed-integer linear program which can be solved to global optimality by classical integer programming solvers. Additionally, a heuristic solution algorithm is presented and we study the model under data uncertainty, applying a two-stage robust optimization approach. We implemented our methods on random and real datasets and show that the heuristic version of the BDNN outperforms classical deep neural networks on the Breast Cancer Wisconsin dataset while performing worse on random data.