Interpreting FCDNNs via RG on Exponential Family
This work offers a theoretical foundation for interpreting deep learning via statistical physics, but it is limited to specific data distributions and fully connected architectures, making it incremental.
The authors prove that for fully connected deep neural networks trained on exponential family data, the trained network's feature layer parameters correspond to fixed points of the renormalization group transformation, establishing an equivalence between DNN training and RG calculation. This provides a theoretical explanation for DNNs' feature extraction capabilities.
We consider establishing the interpretability theory of deep learning through constructing a corresponding relationship between the renormalization group (RG) method in statistical physics and the training process of deep neural networks (DNNs). We have proved the constructed relationship using the one-dimensional Ising model as the input data. In this paper we generalize our results to the case of continuous input data, which is a necessary preparation for applying the corresponding framework to real-world data. To be representative, we consider a class of data distribution in the exponential family. We prove that when the parameters of fully connected (FC) DNNs achieve their optimal value after training, the characteristic parameters of the feature layer output of DNNs are equal to the fixed points of the characteristic parameters of input data under RG method for continuous fields. This conclusion shows that the training process of DNNs is equivalent to RG calculation on this kind of data and therefore the network can extract main features from the input data just like RG. Also, the equivalence further validates the correspondence framework we have established, providing an explanation for the outstanding performance of DNNs on real-world data.