Using PCA to Efficiently Represent State Spaces
This addresses the problem of slow convergence in high-dimensional state spaces for reinforcement learning practitioners, but it is incremental as it builds on existing dimensionality reduction techniques.
The paper tackles the curse of dimensionality in reinforcement learning by using PCA to project states onto a low-dimensional manifold, finding that learning in 4 dimensions instead of 9 improves performance and convergence rate in the Mario Benchmarking Domain.
Reinforcement learning algorithms need to deal with the exponential growth of states and actions when exploring optimal control in high-dimensional spaces. This is known as the curse of dimensionality. By projecting the agent's state onto a low-dimensional manifold, we can represent the state space in a smaller and more efficient representation. By using this representation during learning, the agent can converge to a good policy much faster. We test this approach in the Mario Benchmarking Domain. When using dimensionality reduction in Mario, learning converges much faster to a good policy. But, there is a critical convergence-performance trade-off. By projecting onto a low-dimensional manifold, we are ignoring important data. In this paper, we explore this trade-off of convergence and performance. We find that learning in as few as 4 dimensions (instead of 9), we can improve performance past learning in the full dimensional space at a faster convergence rate.