Adaptive Reward Design for Reinforcement Learning
This addresses a bottleneck in reinforcement learning for complex tasks, offering incremental improvements in reward design for uncertain environments.
The paper tackles the problem of sparse rewards in reinforcement learning for tasks specified by Linear Temporal Logic, which fail to encourage subtask completion, by proposing adaptive reward shaping that dynamically updates reward functions. Experimental results show the approach outperforms baselines with earlier convergence, higher expected return, and improved task completion rates.
There is a surge of interest in using formal languages such as Linear Temporal Logic (LTL) to precisely and succinctly specify complex tasks and derive reward functions for Reinforcement Learning (RL). However, existing methods often assign sparse rewards (e.g., giving a reward of 1 only if a task is completed and 0 otherwise). By providing feedback solely upon task completion, these methods fail to encourage successful subtask completion. This is particularly problematic in environments with inherent uncertainty, where task completion may be unreliable despite progress on intermediate goals. To address this limitation, we propose a suite of reward functions that incentivize an RL agent to complete a task specified by an LTL formula as much as possible, and develop an adaptive reward shaping approach that dynamically updates reward functions during the learning process. Experimental results on a range of benchmark RL environments demonstrate that the proposed approach generally outperforms baselines, achieving earlier convergence to a better policy with higher expected return and task completion rate.