Quiet Feature Learning in Algorithmic Tasks
This work addresses the problem of monitoring model training for researchers and practitioners, revealing that substantial representational progress can be hidden beneath flat loss curves, which is incremental but important for diagnostic methods.
The authors trained Transformer language models on algorithmic tasks and found that validation loss remained flat over large compute ranges before abruptly decreasing, with 'quiet features' representing intermediate algorithmic computations being learned during the flat phase. Ablation experiments showed these features are causally necessary for task performance, challenging the use of cross-entropy loss as a proxy for learning.
We train Transformer-based language models on ten foundational algorithmic tasks and observe pronounced phase transitions in their loss curves that deviate from established power-law scaling trends. Over large ranges of compute, the validation loss barely improves, then abruptly decreases. Probing the models' internal representations reveals that quiet features are learned prior to any decrease in task loss. These quiet features represent intermediate algorithmic computations that do not by themselves improve the output loss. Ablation experiments demonstrate that individual quiet features are causally necessary for task performance. Our results demonstrate that substantial representational progress can remain hidden beneath an apparently flat loss curve, challenging the prevailing use of cross-entropy as a proxy for learning and motivating richer diagnostics for monitoring model training.