Kernel Dependence Network
This work addresses the challenge of optimizing network architecture and training in deep learning, offering a novel spectral approach with theoretical insights, though it appears incremental as it builds on existing kernel and manifold methods.
The authors tackled the problem of training deep networks for multi-class classification by proposing a greedy spectral method that learns linear weights to maximize dependence between layer outputs and labels using HSIC, resulting in a network that automatically determines its width and depth with theoretical guarantees for global optimum and generalization.
We propose a greedy strategy to spectrally train a deep network for multi-class classification. Each layer is defined as a composition of linear weights with the feature map of a Gaussian kernel acting as the activation function. At each layer, the linear weights are learned by maximizing the dependence between the layer output and the labels using the Hilbert Schmidt Independence Criterion (HSIC). By constraining the solution space on the Stiefel Manifold, we demonstrate how our network construct (Kernel Dependence Network or KNet) can be solved spectrally while leveraging the eigenvalues to automatically find the width and the depth of the network. We theoretically guarantee the existence of a solution for the global optimum while providing insight into our network's ability to generalize.