Interpreting Basis Path Set in Neural Networks
This work addresses a theoretical gap for researchers in neural network optimization, but it appears incremental as it builds on existing basis path concepts without demonstrating broad practical improvements.
The paper tackles the problem of interpreting the inner mechanism of basis paths in neural networks, showing that the proposed hierarchical algorithm HBPS can find basis path sets by decomposing networks into independent substructures.
Based on basis path set, G-SGD algorithm significantly outperforms conventional SGD algorithm in optimizing neural networks. However, how the inner mechanism of basis paths work remains mysterious. From the aspect of graph theory, this paper defines basis path, investigates structure properties of basis paths in regular fully connected neural network and interprets the graph representation of basis path set. Moreover, we propose hierarchical algorithm HBPS to find basis path set B in fully connected neural network by decomposing the network into several independent and parallel substructures. Algorithm HBPS demands that there doesn't exist shared edges between any two independent substructure paths.