Reinforcement Learning from Bagged Reward
This addresses a challenge in applying RL to real-world scenarios where designing immediate rewards is difficult, but it appears incremental as it builds on existing reward redistribution methods.
The paper tackles the problem of reinforcement learning when only aggregated rewards are provided for sequences of actions, rather than immediate rewards, by formulating it as RL from Bagged Reward (RLBR) and proposing a bidirectional attention mechanism for reward redistribution. The result shows that their method consistently outperforms existing approaches, though no concrete numbers are provided.
In Reinforcement Learning (RL), it is commonly assumed that an immediate reward signal is generated for each action taken by the agent, helping the agent maximize cumulative rewards to obtain the optimal policy. However, in many real-world scenarios, designing immediate reward signals is difficult; instead, agents receive a single reward that is contingent upon a partial sequence or a complete trajectory. In this work, we define this challenging problem as RL from Bagged Reward (RLBR), where sequences of data are treated as bags with non-Markovian bagged rewards, leading to the formulation of Bagged Reward Markov Decision Processes (BRMDPs). Theoretically, we demonstrate that RLBR can be addressed by solving a standard MDP with properly redistributed bagged rewards allocated to each instance within a bag. Empirically, we find that reward redistribution becomes more challenging as the bag length increases, due to reduced informational granularity. Existing reward redistribution methods are insufficient to address these challenges. Therefore, we propose a novel reward redistribution method equipped with a bidirectional attention mechanism, enabling the accurate interpretation of contextual nuances and temporal dependencies within each bag. We experimentally demonstrate that our proposed method consistently outperforms existing approaches.