Is Deep Learning a Renormalization Group Flow?
This work addresses a theoretical gap in understanding deep learning for researchers in physics and machine learning, though it is incremental as it focuses on a single layer.
The paper investigates whether deep learning resembles a renormalization group (RG) flow by training a restricted Boltzmann machine on the Ising model and comparing patterns to RG treatments, finding RG-like patterns in correlators but also highlighting differences.
Although there has been a rapid development of practical applications, theoretical explanations of deep learning are in their infancy. Deep learning performs a sophisticated coarse graining. Since coarse graining is a key ingredient of the renormalization group (RG), RG may provide a useful theoretical framework directly relevant to deep learning. In this study we pursue this possibility. A statistical mechanics model for a magnet, the Ising model, is used to train an unsupervised restricted Boltzmann machine (RBM). The patterns generated by the trained RBM are compared to the configurations generated through an RG treatment of the Ising model. Although we are motivated by the connection between deep learning and RG flow, in this study we focus mainly on comparing a single layer of a deep network to a single step in the RG flow. We argue that correlation functions between hidden and visible neurons are capable of diagnosing RG-like coarse graining. Numerical experiments show the presence of RG-like patterns in correlators computed using the trained RBMs. The observables we consider are also able to exhibit important differences between RG and deep learning.