Neural Optimization Kernel: Towards Robust Deep Learning
This work addresses a foundational theoretical gap in deep learning for researchers, though it appears incremental as it builds on existing kernel methods.
The authors tackled the problem of understanding why deep neural networks generalize well despite over-parameterization by connecting them to a new kernel family called Neural Optimization Kernel (NOK), which provides a generalization bound and serves as a plug-in for robustness against input noise.
Deep neural networks (NN) have achieved great success in many applications. However, why do deep neural networks obtain good generalization at an over-parameterization regime is still unclear. To better understand deep NN, we establish the connection between deep NN and a novel kernel family, i.e., Neural Optimization Kernel (NOK). The architecture of structured approximation of NOK performs monotonic descent updates of implicit regularization problems. We can implicitly choose the regularization problems by employing different activation functions, e.g., ReLU, max pooling, and soft-thresholding. We further establish a new generalization bound of our deep structured approximated NOK architecture. Our unsupervised structured approximated NOK block can serve as a simple plug-in of popular backbones for a good generalization against input noise.