Globally Optimal Training of Neural Networks with Threshold Activation Functions
This work addresses a key bottleneck for deploying efficient and interpretable neural networks in hardware, though it is incremental as it builds on convex optimization techniques like LASSO.
The paper tackles the problem of training neural networks with threshold activation functions, which are non-differentiable and thus incompatible with gradient-based methods, by showing that regularized training can be reformulated as a convex optimization problem under certain conditions, such as when the last hidden layer width exceeds a threshold or the dataset is shatterable, and validates this with numerical experiments.
Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable and resembles that of biological neurons. However, traditional gradient based algorithms such as Gradient Descent cannot be used to train the parameters of neural networks with threshold activations since the activation function has zero gradient except at a single non-differentiable point. To this end, we study weight decay regularized training problems of deep neural networks with threshold activations. We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold. We also derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network. We corroborate our theoretical results with various numerical experiments.