LogAvgExp Provides a Principled and Performant Global Pooling Operator
This work addresses a fundamental component in neural network design for researchers and practitioners in computer vision, offering a principled alternative to standard pooling methods.
The paper tackled the problem of improving pooling operations in neural networks by proposing LogAvgExp as a theoretically justified global pooling operator, which smoothly transitions between max and mean pooling via a temperature parameter and was experimentally validated across various deep learning architectures for computer vision.
We seek to improve the pooling operation in neural networks, by applying a more theoretically justified operator. We demonstrate that LogSumExp provides a natural OR operator for logits. When one corrects for the number of elements inside the pooling operator, this becomes $\text{LogAvgExp} := \log(\text{mean}(\exp(x)))$. By introducing a single temperature parameter, LogAvgExp smoothly transitions from the max of its operands to the mean (found at the limiting cases $t \to 0^+$ and $t \to +\infty$). We experimentally tested LogAvgExp, both with and without a learnable temperature parameter, in a variety of deep neural network architectures for computer vision.