Partially Observable Reinforcement Learning with Memory Traces
This addresses the problem of computational intractability in partially observable reinforcement learning for researchers and practitioners, representing an incremental improvement over existing window-based methods.
The paper tackles the computational challenge of partially observable reinforcement learning by introducing memory traces, which are compact representations of observation histories using exponential moving averages. The authors prove sample complexity bounds for offline on-policy evaluation and demonstrate that memory traces can be significantly more sample efficient than window-based approaches in certain environments.
Partially observable environments present a considerable computational challenge in reinforcement learning due to the need to consider long histories. Learning with a finite window of observations quickly becomes intractable as the window length grows. In this work, we introduce memory traces. Inspired by eligibility traces, these are compact representations of the history of observations in the form of exponential moving averages. We prove sample complexity bounds for the problem of offline on-policy evaluation that quantify the return errors achieved with memory traces for the class of Lipschitz continuous value estimates. We establish a close connection to the window approach, and demonstrate that, in certain environments, learning with memory traces is significantly more sample efficient. Finally, we underline the effectiveness of memory traces empirically in online reinforcement learning experiments for both value prediction and control.