Scale redundancy and soft gauge fixing in positively homogeneous neural networks
This work addresses optimization conditioning in neural networks for researchers, offering a gauge-theoretic approach to mitigate scale drift, but it is incremental as it builds on known symmetries without introducing a new paradigm.
The paper tackled the problem of continuous reparametrization symmetry in neural networks with positively homogeneous activations, which causes scale redundancy and optimization issues; by introducing a soft gauge-fixing penalty inspired by field theory, they analytically showed it induces dissipative relaxation and experimentally expanded the stable learning-rate regime and suppressed scale drift without affecting expressivity.
Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which the input--output function is invariant. We interpret this symmetry as a gauge redundancy and introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions. Inspired by gauge fixing in field theory, we introduce a soft orbit-selection (norm-balancing) functional acting only on redundant scale coordinates. We show analytically that it induces dissipative relaxation of imbalance modes to preserve the realized function. In controlled experiments, this orbit-selection penalty expands the stable learning-rate regime and suppresses scale drift without changing expressivity. These results establish a structural link between gauge-orbit geometry and optimization conditioning, providing a concrete connection between gauge-theoretic concepts and machine learning.