On the space of coefficients of a Feed Forward Neural Network
This work addresses a foundational issue in neural network theory by formalizing equivalence, which is incremental but provides a mathematical framework for understanding parameter redundancy.
The paper tackles the problem of characterizing equivalent neural networks with different parameters that produce the same function, proving that for networks with piece-wise linear activations, the space of equivalent coefficients forms a semialgebraic set.
We define and establish the conditions for `equivalent neural networks' - neural networks with different weights, biases, and threshold functions that result in the same associated function. We prove that given a neural network $\mathcal{N}$ with piece-wise linear activation, the space of coefficients describing all equivalent neural networks is given by a semialgebraic set. This result is obtained by studying different representations of a given piece-wise linear function using the Tarski-Seidenberg theorem.