Multi-step Predictive Coding Leads To Simplicity Bias
This provides theoretical grounding for when predictive coding yields interpretable representations, addressing a gap in understanding for researchers in representation learning.
The paper investigates how prediction horizon and network depth affect the emergence of structured representations in predictive coding, showing that deep networks with multi-step prediction consistently recover underlying latent structures across various settings.
Predictive coding is a framework for understanding the formation of low-dimensional internal representations mirroring the environment's latent structure. The conditions under which such representations emerge remain unclear. In this work, we investigate how the prediction horizon and network depth shape the solutions of predictive coding tasks. Using a minimal abstract setting inspired by prior work, we show empirically and theoretically that sufficiently deep networks trained with multi-step prediction horizons consistently recover the underlying latent structure, a phenomenon explained through the Ordinary Least Squares estimator structure and biases in learning dynamics. We then extend these insights to nonlinear networks and complex datasets, including piecewise linear functions, MNIST, multiple latent states and higher dimensional state geometries. Our results provide a principled understanding of when and why predictive coding induces structured representations, bridging the gap between empirical observations and theoretical foundations.