The In-Sample Softmax for Offline Reinforcement Learning
This addresses a key bottleneck in offline RL for improving policy extraction from limited datasets, though it is incremental as it builds on existing in-sample max approximations.
The paper tackles the problem of overestimation and divergence in offline reinforcement learning due to insufficient action coverage by proposing an in-sample softmax approach that only uses actions well-covered by the dataset. The result is an In-Sample Actor-Critic method that shows consistent improvement or comparability to existing methods, with convergence guarantees and suitability for fine-tuning.
Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an \emph{in-sample} max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample \emph{softmax} using only actions in the dataset. We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well-suited to fine-tuning.