Understanding the Distributions of Aggregation Layers in Deep Neural Networks
This work addresses a foundational issue in deep learning by providing insights into aggregation layers, which are crucial for feature consolidation and performance, though it appears incremental in advancing theoretical understanding.
The paper tackled the problem of understanding the distributions of aggregation layers in deep neural networks, proposing a novel mathematical formulation to analytically model these distributions and predict KL-divergence, with experimental verification across various classification tasks and datasets.
The process of aggregation is ubiquitous in almost all deep nets models. It functions as an important mechanism for consolidating deep features into a more compact representation, whilst increasing robustness to overfitting and providing spatial invariance in deep nets. In particular, the proximity of global aggregation layers to the output layers of DNNs mean that aggregated features have a direct influence on the performance of a deep net. A better understanding of this relationship can be obtained using information theoretic methods. However, this requires the knowledge of the distributions of the activations of aggregation layers. To achieve this, we propose a novel mathematical formulation for analytically modelling the probability distributions of output values of layers involved with deep feature aggregation. An important outcome is our ability to analytically predict the KL-divergence of output nodes in a DNN. We also experimentally verify our theoretical predictions against empirical observations across a range of different classification tasks and datasets.