Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow
This work provides insights into feature extraction in neural networks for researchers in theoretical machine learning and statistical physics, though it is incremental as it builds on known analogies between DNNs and RG.
The paper investigates the relationship between deep neural networks and renormalization group (RG) flow by training a Restricted Boltzmann Machine (RBM) on Ising model spin configurations across temperatures from 0 to 6, showing that the RBM generates a flow that approaches the critical temperature Tc=2.27, which is opposite to typical RG flow.
Theoretical understanding of how deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse-graining. It reminds us of the basic concept of renormalization group (RG) in statistical physics. In order to explore possible relations between DNN and RG, we use the Restricted Boltzmann machine (RBM) applied to Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from $T=0$ to $T=6$ generates a flow along which the temperature approaches the critical value $T_c=2.27$. This behavior is opposite to the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards $T_c$ and how the RBM learns to extract features of spin configurations.