Criticality & Deep Learning II: Momentum Renormalisation Group
This work addresses the challenge of achieving scale invariance in deep learning, which could enhance model robustness and generalization, though it appears incremental as it builds on existing critical systems and regularization concepts.
The authors tackled the problem of inducing scale invariance in deep learning systems by developing a novel mechanism using inhomogeneous polynomial regularization, mapping the setup to a field theory and applying the Renormalisation Group in momentum space to derive critical regularization conditions.
Guided by critical systems found in nature we develop a novel mechanism consisting of inhomogeneous polynomial regularisation via which we can induce scale invariance in deep learning systems. Technically, we map our deep learning (DL) setup to a genuine field theory, on which we act with the Renormalisation Group (RG) in momentum space and produce the flow equations of the couplings; those are translated to constraints and consequently interpreted as "critical regularisation" conditions in the optimiser; the resulting equations hence prove to be sufficient conditions for - and serve as an elegant and simple mechanism to induce scale invariance in any deep learning setup.