The Order Is The Message
This reveals a critical, previously overlooked channel for information transfer in machine learning that could impact training efficiency and safety, with broad implications for all of ML/AI.
The paper tackled the problem of how example ordering affects learning in modular arithmetic, finding that specific fixed-order strategies achieved 99.5% test accuracy with only 0.3% of the input space, while IID ordering failed at 0.30% accuracy, and adversarial ordering prevented learning entirely.
In a controlled experiment on modular arithmetic ($p = 9973$), varying only example ordering while holding all else constant, two fixed-ordering strategies achieve 99.5\% test accuracy by epochs 487 and 659 respectively from a training set comprising 0.3\% of the input space, well below established sample complexity lower bounds for this task under IID ordering. The IID baseline achieves 0.30\% after 5{,}000 epochs from identical data. An adversarially structured ordering suppresses learning entirely. The generalizing model reliably constructs a Fourier representation whose fundamental frequency is the Fourier dual of the ordering structure, encoding information present in no individual training example, with the same fundamental emerging across all seeds tested regardless of initialization or training set composition. We discuss implications for training efficiency, the reinterpretation of grokking, and the safety risks of a channel that evades all content-level auditing.