Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning
This addresses the need for more stable and efficient training in reinforcement learning, though it is incremental as it builds on existing RL methods without a paradigm shift.
The paper tackles the problem of backpropagation in reinforcement learning, which requires storing activations and can cause gradient issues, by proposing a method that trains each layer locally during the forward pass using pairwise distance matching, achieving competitive performance and improved stability in RL benchmarks.
Training neural networks with reinforcement learning (RL) typically relies on backpropagation (BP), necessitating storage of activations from the forward pass for subsequent backward updates. Furthermore, backpropagating error signals through multiple layers often leads to vanishing or exploding gradients, which can degrade learning performance and stability. We propose a novel approach that trains each layer of the neural network using local signals during the forward pass in RL settings. Our approach introduces local, layer-wise losses leveraging the principle of matching pairwise distances from multi-dimensional scaling, enhanced with optional reward-driven guidance. This method allows each hidden layer to be trained using local signals computed during forward propagation, thus eliminating the need for backward passes and storing intermediate activations. Our experiments, conducted with policy gradient methods across common RL benchmarks, demonstrate that this backpropagation-free method achieves competitive performance compared to their classical BP-based counterpart. Additionally, the proposed method enhances stability and consistency within and across runs, and improves performance especially in challenging environments.