Reducing Parameter Space for Neural Network Training
This work addresses the challenge of optimizing neural network training by reducing the search space, which is an incremental improvement for machine learning practitioners.
The authors tackled the problem of neural network training by showing that for networks with ReLU or binary activations, training can be performed in a reduced parameter space where weights are on the unit sphere and thresholds in a bounded interval, leading to improved training performance as demonstrated in numerical examples.
For neural networks (NNs) with rectified linear unit (ReLU) or binary activation functions, we show that their training can be accomplished in a reduced parameter space. Specifically, the weights in each neuron can be trained on the unit sphere, as opposed to the entire space, and the threshold can be trained in a bounded interval, as opposed to the real line. We show that the NNs in the reduced parameter space are mathematically equivalent to the standard NNs with parameters in the whole space. The reduced parameter space shall facilitate the optimization procedure for the network training, as the search space becomes (much) smaller. We demonstrate the improved training performance using numerical examples.