Deep Neural Nets with Interpolating Function as Output Activation
This work addresses generalization issues in deep learning for practitioners, though it appears incremental as it modifies an existing component rather than introducing a new paradigm.
The authors tackled the problem of insufficient training data and poor generalization in deep neural networks by replacing the softmax output activation with a novel interpolating function, resulting in significant improvements in generalization accuracy across various networks.
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and code will be made publicly available.