Half-Layered Neural Networks
This is an incremental improvement for neural network efficiency, potentially benefiting applications with limited computational resources.
The paper tackles the problem of reducing the number of trainable parameters in neural networks by introducing a 'half' layer with partially randomized weights, achieving reasonable accuracy on MNIST and FashionMNIST datasets.
We propose a ``half'' layer of hidden units that has some of its weights randomly set and some of them trained. A half unit is composed of two stages: First, it takes a weighted sum of its inputs with fixed random weights, and second, the total activation is multiplied and then translated using two modifiable weights, before the result is passed through a nonlinearity. The number of modifiable weights of each hidden unit is thus two and does not depend on the fan-in. We show how such half units can be used in the first or any later layer in a deep network, possibly following convolutional layers. Our experiments on MNIST and FashionMNIST data sets indicate the promise of half layers, where we can achieve reasonable accuracy with a reduced number of parameters due to the regularizing effect of the randomized connections.