On the inductive bias of infinite-depth ResNets and the bottleneck rank
arXiv:2501.19149v12 citationsh-index: 8
Originality Incremental advance
AI Analysis
This provides theoretical insights into the behavior of deep ResNets, which is incremental for understanding neural network optimization in machine learning.
The paper investigates the inductive bias of infinite-depth ResNets, showing that it lies between minimizing nuclear norm and rank, leading to a bias towards minimizing bottleneck rank in nonlinear ResNets with proper hyperparameters.
We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that, with appropriate hyperparameters, deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.