Action Space Shaping in Deep Reinforcement Learning
This work addresses the challenge of designing effective action spaces for RL practitioners, but it is incremental as it provides systematic insights rather than a new method.
The paper tackled the problem of action space modifications in deep reinforcement learning for video-game environments, which are currently based on intuition, by conducting extensive experiments to show that domain-specific removal of actions and discretization of continuous actions are crucial for successful learning.
Reinforcement learning (RL) has been successful in training agents in various learning environments, including video-games. However, such work modifies and shrinks the action space from the game's original. This is to avoid trying "pointless" actions and to ease the implementation. Currently, this is mostly done based on intuition, with little systematic research supporting the design decisions. In this work, we aim to gain insight on these action space modifications by conducting extensive experiments in video-game environments. Our results show how domain-specific removal of actions and discretization of continuous actions can be crucial for successful learning. With these insights, we hope to ease the use of RL in new environments, by clarifying what action-spaces are easy to learn.