Generalized Back-Stepping Experience Replay in Sparse-Reward Environments
This is an incremental improvement for reinforcement learning in sparse-reward environments like maze navigation.
The paper tackles the limitation of Back-stepping Experience Replay (BER) in sparse-reward environments by proposing Generalized BER (GBER), which introduces relabeling and diverse sampling strategies. Experimental results show GBER significantly boosts performance and stability in maze navigation tasks, particularly in structurally symmetric environments.
Back-stepping experience replay (BER) is a reinforcement learning technique that can accelerate learning efficiency in reversible environments. BER trains an agent with generated back-stepping transitions of collected experiences and normal forward transitions. However, the original algorithm is designed for a dense-reward environment that does not require complex exploration, limiting the BER technique to demonstrate its full potential. Herein, we propose an enhanced version of BER called Generalized BER (GBER), which extends the original algorithm to sparse-reward environments, particularly those with complex structures that require the agent to explore. GBER improves the performance of BER by introducing relabeling mechanism and applying diverse sampling strategies. We evaluate our modified version, which is based on a goal-conditioned deep deterministic policy gradient offline learning algorithm, across various maze navigation environments. The experimental results indicate that the GBER algorithm can significantly boost the performance and stability of the baseline algorithm in various sparse-reward environments, especially those with highly structural symmetricity.