Discrete and Continuous Action Representation for Practical RL in Video Games
This work addresses the practical constraints faced by the video game industry in implementing reinforcement learning, though it appears incremental as it builds on existing SAC methods.
The authors tackled the problem of applying reinforcement learning under practical constraints in video games by proposing Hybrid SAC, an extension of Soft Actor-Critic that handles discrete, continuous, and parameterized actions. They demonstrated that Hybrid SAC successfully solved a high-speed driving task in a game and was competitive with state-of-the-art methods on benchmark tasks.
While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.