Unveiling the Training Dynamics of ReLU Networks through a Linear Lens
This work provides a new interpretability tool for deep learning researchers, offering insights into representation learning dynamics, though it is incremental in advancing theoretical understanding.
The authors tackled the challenge of understanding the internal learning mechanisms of ReLU neural networks by proposing an analytical framework that recasts them into equivalent single-layer linear models with input-dependent effective weights, demonstrating that these weights converge for same-class samples and diverge for different classes during training.
Deep neural networks, particularly those employing Rectified Linear Units (ReLU), are often perceived as complex, high-dimensional, non-linear systems. This complexity poses a significant challenge to understanding their internal learning mechanisms. In this work, we propose a novel analytical framework that recasts a multi-layer ReLU network into an equivalent single-layer linear model with input-dependent "effective weights". For any given input sample, the activation pattern of ReLU units creates a unique computational path, effectively zeroing out a subset of weights in the network. By composing the active weights across all layers, we can derive an effective weight matrix, $W_{\text{eff}}(x)$, that maps the input directly to the output for that specific sample. We posit that the evolution of these effective weights reveals fundamental principles of representation learning. Our work demonstrates that as training progresses, the effective weights corresponding to samples from the same class converge, while those from different classes diverge. By tracking the trajectories of these sample-wise effective weights, we provide a new lens through which to interpret the formation of class-specific decision boundaries and the emergence of semantic representations within the network.