Expressivity of congruence-based architectures for DNNs on positive-definite matrices
Theoretical insight for researchers designing deep neural networks on symmetric positive-definite matrices, revealing a fundamental limitation of a widely-used architecture.
The paper shows that the semi-orthogonality constraint on weight matrices in congruence-based layers (used in SPDNet) limits expressivity, causing the architecture to collapse to a one-hidden-layer equivalent for certain activation functions. This is due to a loss of spectral diversity, proven via Poincaré's separation theorem.
This work studies neural architectures for classifying symmetric positive-definite matrices, focusing on congruence-like layers, in which the input matrix is multiplied on the left and right by a (possibly rectangular) weight matrix $W$ and its transpose. Such layers lie at the core of the celebrated SPDNet and have also been employed independently for dimensionality reduction on positive-definite data. We show that the (semi)-orthogonality constraint commonly imposed on $W$ limits the expressivity of these layers: for certain activation functions, the resulting architecture collapses to a one-hidden-layer equivalent. This lack of expressivity follows from a loss of spectral diversity in congruence-like layers for semi-orthogonal $W$ and is a direct consequence of Poincaré's separation theorem. We then examine the choice of the final classifier, comparing several Riemannian classifiers and discussing their compatibility with the feature maps produced by congruence-like layers.