Novelty-based Sample Reuse for Continuous Robotics Control
This addresses the time-consuming nature of data collection in complex robotic simulations and real-world applications, though it is an incremental improvement over existing methods.
The paper tackles the problem of inefficient sample utilization in reinforcement learning for robotics control by proposing Novelty-guided Sample Reuse (NSR), which improves convergence and success rates without significantly increasing time consumption.
In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing time consumption. Our code is publicly available at https://github.com/ppksigs/NSR-DDPG-HER.