Choquet regularization for reinforcement learning
This work addresses exploration management in reinforcement learning, offering a novel regularization approach that is incremental, building on prior entropy-based methods.
The authors tackled the problem of managing exploration in reinforcement learning by proposing Choquet regularizers, which replace differential entropy in continuous-time entropy-regularized RL, and derived explicit optimal distributions for linear-quadratic cases, connecting them to common exploratory samplers like ε-greedy and Gaussian.
We propose \emph{Choquet regularizers} to measure and manage the level of exploration for reinforcement learning (RL), and reformulate the continuous-time entropy-regularized RL problem of Wang et al. (2020, JMLR, 21(198)) in which we replace the differential entropy used for regularization with a Choquet regularizer. We derive the Hamilton--Jacobi--Bellman equation of the problem, and solve it explicitly in the linear--quadratic (LQ) case via maximizing statically a mean--variance constrained Choquet regularizer. Under the LQ setting, we derive explicit optimal distributions for several specific Choquet regularizers, and conversely identify the Choquet regularizers that generate a number of broadly used exploratory samplers such as $ε$-greedy, exponential, uniform and Gaussian.