An Empirical Study on the Power of Future Prediction in Partially Observable Environments
This addresses the challenge of learning good representations in partially observable RL environments, though it appears incremental as it builds on existing self-predictive auxiliary task methods.
The study investigated whether future prediction alone can produce effective history representations for reinforcement learning in partially observable environments with long-term dependencies, finding that it consistently enabled strong RL performance across different network architectures.
Learning good representations of historical contexts is one of the core challenges of reinforcement learning (RL) in partially observable environments. While self-predictive auxiliary tasks have been shown to improve performance in fully observed settings, their role in partial observability remains underexplored. In this empirical study, we examine the effectiveness of self-predictive representation learning via future prediction, i.e., predicting next-step observations as an auxiliary task for learning history representations, especially in environments with long-term dependencies. We test the hypothesis that future prediction alone can produce representations that enable strong RL performance. To evaluate this, we introduce $\texttt{DRL}^2$, an approach that explicitly decouples representation learning from reinforcement learning, and compare this approach to end-to-end training across multiple benchmarks requiring long-term memory. Our findings provide evidence that this hypothesis holds across different network architectures, reinforcing the idea that future prediction performance serves as a reliable indicator of representation quality and contributes to improved RL performance.