Non-identifiability distinguishes Neural Networks among Parametric Models
This work addresses a foundational issue in machine learning by clarifying theoretical differences between neural networks and parametric models, though it is incremental as it builds on existing identifiability concepts.
The paper tackles the problem of distinguishing neural networks from traditional statistical models by proving that neural networks always learn a nontrivial relationship between variables if one exists, whereas other smooth parametric models can fail under identifiability conditions, learning only the constant predictor.
One of the enduring problems surrounding neural networks is to identify the factors that differentiate them from traditional statistical models. We prove a pair of results which distinguish feedforward neural networks among parametric models at the population level, for regression tasks. Firstly, we prove that for any pair of random variables $(X,Y)$, neural networks always learn a nontrivial relationship between $X$ and $Y$, if one exists. Secondly, we prove that for reasonable smooth parametric models, under local and global identifiability conditions, there exists a nontrivial $(X,Y)$ pair for which the parametric model learns the constant predictor $\mathbb{E}[Y]$. Together, our results suggest that a lack of identifiability distinguishes neural networks among the class of smooth parametric models.