Exploring epoch-dependent stochastic residual networks
This is an incremental improvement for neural network training methods.
The paper tackled the problem of training stochastic residual networks by introducing epoch-dependent probability distributions for layer activation, starting with higher bypass probabilities for deeper layers and increasing activation over time. Preliminary results were mixed but indicated potential for further investigation.
The recently proposed stochastic residual networks selectively activate or bypass the layers during training, based on independent stochastic choices, each of which following a probability distribution that is fixed in advance. In this paper we present a first exploration on the use of an epoch-dependent distribution, starting with a higher probability of bypassing deeper layers and then activating them more frequently as training progresses. Preliminary results are mixed, yet they show some potential of adding an epoch-dependent management of distributions, worth of further investigation.