Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks
This work addresses the structure of parameter spaces in neural networks, which is foundational for understanding learning processes, but it is incremental as it focuses on a specific architecture.
The paper tackled the problem of characterizing the functional equivalence classes of reducible parameters in single-hidden-layer hyperbolic tangent neural networks, showing that these classes are piecewise-linear path-connected sets with a diameter of at most 7 linear segments for parameters with many redundant units.
Understanding the learning process of artificial neural networks requires clarifying the structure of the parameter space within which learning takes place. A neural network parameter's functional equivalence class is the set of parameters implementing the same input--output function. For many architectures, almost all parameters have a simple and well-documented functional equivalence class. However, there is also a vanishing minority of reducible parameters, with richer functional equivalence classes caused by redundancies among the network's units. In this paper, we give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture. We show that such functional equivalence classes are piecewise-linear path-connected sets, and that for parameters with a majority of redundant units, the sets have a diameter of at most 7 linear segments.