Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment
This work addresses efficiency and scalability issues in biologically inspired neural network training, offering potential for custom hardware implementations, though it is incremental as it builds on existing predictive coding and feedback alignment methods.
The paper tackled the limitations of predictive coding networks, such as slow error propagation and vanishing updates in early layers, by proposing direct Kolen-Pollack predictive coding (DKP-PC), which reduces error propagation time complexity from O(L) to O(1) and achieves comparable or better performance than standard PC.
Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.