Control of Overfitting with Physics
This work provides theoretical insights into generalization properties for researchers in machine learning, but it is incremental as it builds on existing analogies without introducing new methods.
The authors tackled the problem of understanding theoretical justifications for overfitting control in machine learning by drawing analogies from physics and biology. They showed that the Eyring formula from kinetic theory can control overfitting in stochastic gradient Langevin dynamics, and an analogy to predator-prey models explains overfitting reduction in GANs.
While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach - when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and overfitting reduction for GANs.