Effects of the optimisation of the margin distribution on generalisation in deep architectures
This addresses the generalization issue in deep learning for practitioners, but it appears incremental as it adapts a known principle from SVMs to deep architectures.
The paper tackled the problem of improving generalization in deep learning by shifting from margin maximization to minimizing margin variance, proposing the Halfway loss function that reduces Normalized Margin Variance. The result showed competitive performance against Softmax Cross-Entropy on MNIST, smallNORB, and CIFAR-10 datasets, though specific numbers are not provided in the abstract.
Despite being so vital to success of Support Vector Machines, the principle of separating margin maximisation is not used in deep learning. We show that minimisation of margin variance and not maximisation of the margin is more suitable for improving generalisation in deep architectures. We propose the Halfway loss function that minimises the Normalised Margin Variance (NMV) at the output of a deep learning models and evaluate its performance against the Softmax Cross-Entropy loss on the MNIST, smallNORB and CIFAR-10 datasets.