Reward Propagation Using Graph Convolutional Networks
This work addresses the challenge of speeding up learning in reinforcement learning for complex environments, though it is incremental as it builds on existing reward shaping and graph representation learning methods.
The paper tackles the problem of automatically finding potential functions for reward shaping in reinforcement learning by proposing a framework that uses Graph Convolutional Networks to propagate messages from rewarding states, achieving considerable improvements in small and high-dimensional control problems.
Potential-based reward shaping provides an approach for designing good reward functions, with the purpose of speeding up learning. However, automatically finding potential functions for complex environments is a difficult problem (in fact, of the same difficulty as learning a value function from scratch). We propose a new framework for learning potential functions by leveraging ideas from graph representation learning. Our approach relies on Graph Convolutional Networks which we use as a key ingredient in combination with the probabilistic inference view of reinforcement learning. More precisely, we leverage Graph Convolutional Networks to perform message passing from rewarding states. The propagated messages can then be used as potential functions for reward shaping to accelerate learning. We verify empirically that our approach can achieve considerable improvements in both small and high-dimensional control problems.