Decoupling regularization from the action space
This addresses a practical issue in regularized RL for applications with varying action spaces, such as control and sequence design, but is incremental as it builds on existing regularization frameworks.
The paper tackles the problem of over-regularization in entropy-regularized reinforcement learning when action spaces vary, showing that standard methods are affected by changes in the number of actions. It introduces static and dynamic temperature selection solutions, which improve performance on tasks like the DeepMind control suite and biological sequence design.
Regularized reinforcement learning (RL), particularly the entropy-regularized kind, has gained traction in optimal control and inverse RL. While standard unregularized RL methods remain unaffected by changes in the number of actions, we show that it can severely impact their regularized counterparts. This paper demonstrates the importance of decoupling the regularizer from the action space: that is, to maintain a consistent level of regularization regardless of how many actions are involved to avoid over-regularization. Whereas the problem can be avoided by introducing a task-specific temperature parameter, it is often undesirable and cannot solve the problem when action spaces are state-dependent. In the state-dependent action context, different states with varying action spaces are regularized inconsistently. We introduce two solutions: a static temperature selection approach and a dynamic counterpart, universally applicable where this problem arises. Implementing these changes improves performance on the DeepMind control suite in static and dynamic temperature regimes and a biological sequence design task.