Learning distinct features helps, provably
This work addresses a foundational issue in machine learning by providing theoretical insights into feature learning, though it is incremental as it builds on existing complexity analysis.
The paper tackles the problem of understanding how feature diversity in neural networks affects generalization, proving that more distinct hidden-layer features lead to better generalization performance, with theoretical bounds derived using Rademacher complexity.
We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L_2$-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network's units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.