Skill or Luck? Return Decomposition via Advantage Functions
This addresses sample efficiency in reinforcement learning for researchers and practitioners, offering an incremental improvement over existing off-policy methods.
The paper tackles the problem of learning from off-policy data in reinforcement learning by decomposing returns into skill (agent-controlled) and luck (external) components using advantage functions, and introduces Off-policy DAE, which avoids importance sampling or truncation, showing it can speed up learning and prevent suboptimal performance in MinAtar environments.
Learning from off-policy data is essential for sample-efficient reinforcement learning. In the present work, we build on the insight that the advantage function can be understood as the causal effect of an action on the return, and show that this allows us to decompose the return of a trajectory into parts caused by the agent's actions (skill) and parts outside of the agent's control (luck). Furthermore, this decomposition enables us to naturally extend Direct Advantage Estimation (DAE) to off-policy settings (Off-policy DAE). The resulting method can learn from off-policy trajectories without relying on importance sampling techniques or truncating off-policy actions. We draw connections between Off-policy DAE and previous methods to demonstrate how it can speed up learning and when the proposed off-policy corrections are important. Finally, we use the MinAtar environments to illustrate how ignoring off-policy corrections can lead to suboptimal policy optimization performance.