Understanding the Dynamics of DNNs Using Graph Modularity
This provides theoretical guidance for layer pruning in DNNs, though it's an incremental step in understanding network dynamics.
The paper tackles the problem of understanding feature representation dynamics in deep neural networks by modeling class separation as community evolution in dynamic graphs and using modularity to quantify this evolution. They demonstrate that degradation/plateau patterns in modularity curves identify redundant layers that can be pruned with minimal performance impact.
There are good arguments to support the claim that deep neural networks (DNNs) capture better feature representations than the previous hand-crafted feature engineering, which leads to a significant performance improvement. In this paper, we move a tiny step towards understanding the dynamics of feature representations over layers. Specifically, we model the process of class separation of intermediate representations in pre-trained DNNs as the evolution of communities in dynamic graphs. Then, we introduce modularity, a generic metric in graph theory, to quantify the evolution of communities. In the preliminary experiment, we find that modularity roughly tends to increase as the layer goes deeper and the degradation and plateau arise when the model complexity is great relative to the dataset. Through an asymptotic analysis, we prove that modularity can be broadly used for different applications. For example, modularity provides new insights to quantify the difference between feature representations. More crucially, we demonstrate that the degradation and plateau in modularity curves represent redundant layers in DNNs and can be pruned with minimal impact on performance, which provides theoretical guidance for layer pruning. Our code is available at https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.