Exploring the Properties and Evolution of Neural Network Eigenspaces during Training
This work provides incremental insights into neural network behavior during training, aiding researchers in optimizing model parameterization for machine learning tasks.
The study investigated how problem difficulty and neural network capacity interact to affect predictive performance, revealing that these factors have an antagonistic effect, which can help detect over- and under-parameterization for specific tasks, with saturation patterns converging early in training to enable faster analysis cycles.
In this work we explore the information processing inside neural networks using logistic regression probes \cite{probes} and the saturation metric \cite{featurespace_saturation}. We show that problem difficulty and neural network capacity affect the predictive performance in an antagonistic manner, opening the possibility of detecting over- and under-parameterization of neural networks for a given task. We further show that the observed effects are independent from previously reported pathological patterns like the ``tail pattern'' described in \cite{featurespace_saturation}. Finally we are able to show that saturation patterns converge early during training, allowing for a quicker cycle time during analysis