Tactical Reward Shaping: Bypassing Reinforcement Learning with Strategy-Based Goals
This addresses inefficiencies in DRL for multi-agent competitive environments, offering a method to bypass extensive learning, though it is incremental as it adapts existing algorithms with new reward structures.
The paper tackles the problem of inefficient reinforcement learning in competitive games by proposing a strategy-based reward shaping approach, demonstrating that geometric-strategic goals enable faster convergence and outperform traditional DQL by orders of magnitude in the ICRA-DJI RoboMaster AI Challenge.
Deep Reinforcement Learning (DRL) has shown its promising capabilities to learn optimal policies directly from trial and error. However, learning can be hindered if the goal of the learning, defined by the reward function, is "not optimal". We demonstrate that by setting the goal/target of competition in a counter-intuitive but intelligent way, instead of heuristically trying solutions through many hours the DRL simulation can quickly converge into a winning strategy. The ICRA-DJI RoboMaster AI Challenge is a game of cooperation and competition between robots in a partially observable environment, quite similar to the Counter-Strike game. Unlike the traditional approach to games, where the reward is given at winning the match or hitting the enemy, our DRL algorithm rewards our robots when in a geometric-strategic advantage, which implicitly increases the winning chances. Furthermore, we use Deep Q Learning (DQL) to generate multi-agent paths for moving, which improves the cooperation between two robots by avoiding the collision. Finally, we implement a variant A* algorithm with the same implicit geometric goal as DQL and compare results. We conclude that a well-set goal can put in question the need for learning algorithms, with geometric-based searches outperforming DQL in many orders of magnitude.