On shallow planning under partial observability
This work addresses the problem of designing effective reinforcement learning agents for real-world scenarios with partial observability, but it is incremental as it builds on existing discount factor analysis.
The paper investigates how the discount factor in reinforcement learning affects the bias-variance trade-off in Markov Decision Processes, finding that a shorter planning horizon can be beneficial, particularly under partial observability.
Formulating a real-world problem under the Reinforcement Learning framework involves non-trivial design choices, such as selecting a discount factor for the learning objective (discounted cumulative rewards), which articulates the planning horizon of the agent. This work investigates the impact of the discount factor on the bias-variance trade-off given structural parameters of the underlying Markov Decision Process. Our results support the idea that a shorter planning horizon might be beneficial, especially under partial observability.