DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learning
This work addresses a scalability bottleneck for researchers and practitioners using default representations in RL, though it is incremental as it builds on prior methods.
The paper tackles the computational inefficiency of approximating the principal eigenvector of the default representation in reinforcement learning by deriving a direct objective for neural network approximation, and empirically shows its effectiveness in various environments for applications like reward shaping.
In computational reinforcement learning, the default representation (DR) and its principal eigenvector have been shown to be effective for a wide variety of applications, including reward shaping, count-based exploration, option discovery, and transfer. However, in prior investigations, the eigenvectors of the DR were computed by first approximating the DR matrix, and then performing an eigendecomposition. This procedure is computationally expensive and does not scale to high-dimensional spaces. In this paper, we derive an objective for directly approximating the principal eigenvector of the DR with a neural network. We empirically demonstrate the effectiveness of the objective in a number of environments, and apply the learned eigenvectors for reward shaping.