Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning
This work provides the first theoretical guarantees for PC-trained deep models, positioning it as a biologically plausible alternative to backpropagation for researchers in machine learning and neuroscience.
The paper tackles the problem of connecting predictive coding (PC), a biologically inspired local learning rule, with the minimum description length (MDL) principle in deep learning, proving that layerwise PC performs block-coordinate descent on the MDL objective and deriving a generalization bound of the form $R(θ) \le \hat{R}(θ) + \frac{L(θ)}{N}$.
We present the first theoretical framework that connects predictive coding (PC), a biologically inspired local learning rule, with the minimum description length (MDL) principle in deep networks. We prove that layerwise PC performs block-coordinate descent on the MDL two-part code objective, thereby jointly minimizing empirical risk and model complexity. Using Hoeffding's inequality and a prefix-code prior, we derive a novel generalization bound of the form $R(θ) \le \hat{R}(θ) + \frac{L(θ)}{N}$, capturing the tradeoff between fit and compression. We further prove that each PC sweep monotonically decreases the empirical two-part codelength, yielding tighter high-probability risk bounds than unconstrained gradient descent. Finally, we show that repeated PC updates converge to a block-coordinate stationary point, providing an approximate MDL-optimal solution. To our knowledge, this is the first result offering formal generalization and convergence guarantees for PC-trained deep models, positioning PC as a theoretically grounded and biologically plausible alternative to backpropagation.