On the effect of pooling on the geometry of representations
This work addresses the theoretical understanding of disentangled representations in machine learning and neuroscience, which is incremental but provides formal insights into existing methods like convolutional neural networks.
The paper investigates how pooling operations in neural networks affect the geometry of learned representations, showing that pooling over input transformations contracts the metric and reduces curvature, with quantitative bounds provided.
In machine learning and neuroscience, certain computational structures and algorithms are known to yield disentangled representations without us understanding why, the most striking examples being perhaps convolutional neural networks and the ventral stream of the visual cortex in humans and primates. As for the latter, it was conjectured that representations may be disentangled by being flattened progressively and at a local scale. An attempt at a formalization of the role of invariance in learning representations was made recently, being referred to as I-theory. In this framework and using the language of differential geometry, we show that pooling over a group of transformations of the input contracts the metric and reduces its curvature, and provide quantitative bounds, in the aim of moving towards a theoretical understanding on how to disentangle representations.