Faster Predictive Coding Networks via Better Initialization
This work addresses the computational inefficiency problem for researchers and practitioners using predictive coding networks, but it is incremental as it focuses on initialization rather than a fundamental change.
The paper tackles the high computational cost of predictive coding networks by proposing a new initialization technique that preserves iterative progress from previous training samples, resulting in substantial improvements in convergence speed and final test loss in supervised and unsupervised settings.
Research aimed at scaling up neuroscience inspired learning algorithms for neural networks is accelerating. Recently, a key research area has been the study of energy-based learning algorithms such as predictive coding, due to their versatility and mathematical grounding. However, the applicability of such methods is held back by the large computational requirements caused by their iterative nature. In this work, we address this problem by showing that the choice of initialization of the neurons in a predictive coding network matters significantly and can notably reduce the required training times. Consequently, we propose a new initialization technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples. Our approach suggests a promising path toward reconciling the disparities between predictive coding and backpropagation in terms of computational efficiency and final performance. In fact, our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.