Training Feedforward Neural Networks with Standard Logistic Activations is Feasible
This addresses a specific training challenge for neural network practitioners, but it is incremental as it focuses on initialization rather than a fundamental breakthrough.
The authors tackled the difficulty of training feedforward neural networks with standard logistic activations by proposing a parameter initialization method derived from information-theoretic analysis, achieving generalization performance comparable to networks with hyperbolic tangent activations.
Training feedforward neural networks with standard logistic activations is considered difficult because of the intrinsic properties of these sigmoidal functions. This work aims at showing that these networks can be trained to achieve generalization performance comparable to those based on hyperbolic tangent activations. The solution consists on applying a set of conditions in parameter initialization, which have been derived from the study of the properties of a single neuron from an information-theoretic perspective. The proposed initialization is validated through an extensive experimental analysis.