Extreme Q-Learning: MaxEnt RL without Entropy
This addresses a key bottleneck in MaxEnt RL for continuous control tasks, offering a novel approach to reduce errors from out-of-distribution actions.
The paper tackles the difficulty of estimating maximal Q-values in continuous action spaces for deep reinforcement learning by introducing Extreme Q-Learning, a framework using Extreme Value Theory to directly model optimal soft-value functions without policy sampling, achieving over 10-point improvements on Franka Kitchen tasks in the D4RL benchmark.
Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our \emph{Extreme Q-Learning} framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by \emph{10+ points} on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks. Visualizations and code can be found on our website at https://div99.github.io/XQL/.