Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry
This provides a theoretical foundation for a common practice in deep learning, addressing a gap in understanding for researchers in computer vision and machine learning, though it is incremental as it clarifies existing methods rather than introducing new ones.
The paper tackles the problem of explaining why Euclidean classifiers work with matrix function normalizations in Global Covariance Pooling by analyzing it through Riemannian geometry, concluding that the mechanism is due to implicit Riemannian classifiers, with empirical validation on fine-grained and large-scale visual classification datasets.
Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations. GCP typically performs classification of the covariance matrices by applying matrix function normalization, such as matrix logarithm or power, followed by a Euclidean classifier. However, covariance matrices inherently lie in a Riemannian manifold, known as the Symmetric Positive Definite (SPD) manifold. The current literature does not provide a satisfactory explanation of why Euclidean classifiers can be applied directly to Riemannian features after the normalization of the matrix power. To mitigate this gap, this paper provides a comprehensive and unified understanding of the matrix logarithm and power from a Riemannian geometry perspective. The underlying mechanism of matrix functions in GCP is interpreted from two perspectives: one based on tangent classifiers (Euclidean classifiers on the tangent space) and the other based on Riemannian classifiers. Via theoretical analysis and empirical validation through extensive experiments on fine-grained and large-scale visual classification datasets, we conclude that the working mechanism of the matrix functions should be attributed to the Riemannian classifiers they implicitly respect. The code is available at https://github.com/GitZH-Chen/RiemGCP.git.