Koopman-based generalization bound: New aspect for full-rank weights
This work addresses the understanding of generalization in neural networks for researchers, focusing on full-rank weights as a complement to existing low-rank analyses, though it is incremental in nature.
The authors tackled the problem of generalization bounds for neural networks with full-rank weight matrices, proposing a new bound based on Koopman operators that is tighter than existing norm-based bounds when condition numbers are small and independent of network width for orthogonal weights.
We propose a new bound for generalization of neural networks using Koopman operators. Whereas most of existing works focus on low-rank weight matrices, we focus on full-rank weight matrices. Our bound is tighter than existing norm-based bounds when the condition numbers of weight matrices are small. Especially, it is completely independent of the width of the network if the weight matrices are orthogonal. Our bound does not contradict to the existing bounds but is a complement to the existing bounds. As supported by several existing empirical results, low-rankness is not the only reason for generalization. Furthermore, our bound can be combined with the existing bounds to obtain a tighter bound. Our result sheds new light on understanding generalization of neural networks with full-rank weight matrices, and it provides a connection between operator-theoretic analysis and generalization of neural networks.