Motif distribution and function of sparse deep neural networks
This work addresses how neural network connectivity relates to function in a domain-specific flight control application, but it is incremental as it builds on existing pruning and motif analysis methods.
The study analyzed the connectivity patterns of 350 sparsely pruned deep neural networks trained for a bio-mechanical flight control task, finding that despite random initialization, enforced sparsity led them to converge to similar motif distributions, suggesting these distributions encode network function.
We characterize the connectivity structure of feed-forward, deep neural networks (DNNs) using network motif theory. To address whether a particular motif distribution is characteristic of the training task, or function of the DNN, we compare the connectivity structure of 350 DNNs trained to simulate a bio-mechanical flight control system with different randomly initialized parameters. We develop and implement algorithms for counting second- and third-order motifs and calculate their significance using their Z-score. The DNNs are trained to solve the inverse problem of the flight dynamics model in Bustamante, et al. (2022) (i.e., predict the controls necessary for controlled flight from the initial and final state-space inputs) and are sparsified through an iterative pruning and retraining algorithm Zahn, et al. (2022). We show that, despite random initialization of network parameters, enforced sparsity causes DNNs to converge to similar connectivity patterns as characterized by their motif distributions. The results suggest how neural network function can be encoded in motif distributions, suggesting a variety of experiments for informing function and control.