Spontaneous symmetry breaking and Goldstone modes for deep information propagation
For deep learning practitioners, this provides a new principle for designing networks that maintain stable information flow, potentially reducing reliance on residual connections or normalization.
The paper shows that deep neural networks with continuous symmetries can support Goldstone-like modes that enable coherent signal propagation across depth and recurrent iterations, improving trainability and long-sequence modeling without architectural stabilizers.
In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.