Network-size independent covering number bounds for deep networks
This work addresses theoretical generalization guarantees for deep networks, but it appears incremental as it builds on existing covering number analyses with a specific simplification.
The authors tackled the problem of bounding covering numbers for deep learning networks by deriving a bound that does not depend on network size, using a method that isolates scaling effects from rotations in linear transformations.
We give a covering number bound for deep learning networks that is independent of the size of the network. The key for the simple analysis is that for linear classifiers, rotating the data doesn't affect the covering number. Thus, we can ignore the rotation part of each layer's linear transformation, and get the covering number bound by concentrating on the scaling part.