Generalized BackPropagation, Étude De Cas: Orthogonality
This work addresses the need for structured constraints in deep learning for applications like filter design and feature extraction, representing a novel method for a known bottleneck rather than a broad paradigm shift.
The paper tackles the problem of incorporating constrained weights, such as orthogonality, into deep networks by extending the backpropagation algorithm using Riemannian geometry and matrix manifold optimization. It introduces a Stiefel layer for orthogonal weights and demonstrates benefits in tasks like unsupervised feature learning and fine-grained image classification, though no concrete numbers are provided.
This paper introduces an extension of the backpropagation algorithm that enables us to have layers with constrained weights in a deep network. In particular, we make use of the Riemannian geometry and optimization techniques on matrix manifolds to step outside of normal practice in training deep networks, equipping the network with structures such as orthogonality or positive definiteness. Based on our development, we make another contribution by introducing the Stiefel layer, a layer with orthogonal weights. Among various applications, Stiefel layers can be used to design orthogonal filter banks, perform dimensionality reduction and feature extraction. We demonstrate the benefits of having orthogonality in deep networks through a broad set of experiments, ranging from unsupervised feature learning to fine-grained image classification.