Pruning-induced phases in fully-connected neural networks: the eumentia, the dementia, and the amentia
This work addresses a fundamental theoretical problem in machine learning by linking neural network pruning to statistical mechanics, offering insights into phase transitions that could inform compression and training strategies.
The study investigated whether pruning induces sharp phase transitions in fully-connected neural networks, identifying three distinct phases (eumentia, dementia, amentia) characterized by power-law scaling of cross-entropy loss with dataset size, with the eumentia-dementia transition showing scale invariance akin to a Berezinskii-Kosterlitz-Thouless-like transition.
Modern neural networks are heavily overparameterized, and pruning, which removes redundant neurons or connections, has emerged as a key approach to compressing them without sacrificing performance. However, while practical pruning methods are well developed, whether pruning induces sharp phase transitions in the neural networks and, if so, to what universality class they belong, remain open questions. To address this, we study fully-connected neural networks trained on MNIST, independently varying the dropout (i.e., removing neurons) rate at both the training and evaluation stages to map the phase diagram. We identify three distinct phases: eumentia (the network learns), dementia (the network has forgotten), and amentia (the network cannot learn), sharply distinguished by the power-law scaling of the cross-entropy loss with the training dataset size. {In the eumentia phase, the algebraic decay of the loss, as documented in the machine learning literature as neural scaling laws, is from the perspective of statistical mechanics the hallmark of quasi-long-range order.} We demonstrate that the transition between the eumentia and dementia phases is accompanied by scale invariance, with a diverging length scale that exhibits hallmarks of a Berezinskii-Kosterlitz-Thouless-like transition; the phase structure is robust across different network widths and depths. Our results establish that dropout-induced pruning provides a concrete setting in which neural network behavior can be understood through the lens of statistical mechanics.