A ghost mechanism: An analytical model of abrupt learning
This work addresses the challenge of understanding abrupt learning mechanisms in neural networks, offering insights for stabilizing training dynamics, though it is incremental as it builds on existing knowledge of learning dynamics.
The authors tackled the problem of abrupt learning in neural networks by introducing a minimal dynamical system that exhibits this behavior through ghost points, showing analytically that a critical learning rate can prevent learning via no-learning zones and oscillatory minima, and confirming these predictions in RNNs with remedies like lowering output confidence and adding sloppy parameters.
\emph{Abrupt learning} is commonly observed in neural networks, where long plateaus in network performance are followed by rapid convergence to a desirable solution. Yet, despite its common occurrence, the complex interplay of task, network architecture, and learning rule has made it difficult to understand the underlying mechanisms. Here, we introduce a minimal dynamical system trained on a delayed-activation task and demonstrate analytically how even a one-dimensional system can exhibit abrupt learning through ghost points rather than bifurcations. Through our toy model, we show that the emergence of a ghost point destabilizes learning dynamics. We identify a critical learning rate that prevents learning through two distinct loss landscape features: a no-learning zone and an oscillatory minimum. Testing these predictions in recurrent neural networks (RNNs), we confirm that ghost points precede abrupt learning and accompany the destabilization of learning. We demonstrate two complementary remedies: lowering the model output confidence prevents the network from getting stuck in no-learning zones, while increasing trainable ranks beyond task requirements (\textit{i.e.}, adding sloppy parameters) provides more stable learning trajectories. Our model reveals a bifurcation-free mechanism for abrupt learning and illustrates the importance of both deliberate uncertainty and redundancy in stabilizing learning dynamics.