You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
This work provides a simple baseline for offline RL, highlighting that current iterative actor-critic methods might be overcomplicated for some tasks, which is incremental but challenges existing approaches.
The paper tackles the problem of offline reinforcement learning by proposing a simple baseline algorithm that performs policy evaluation only once, avoiding complex stabilization schemes. Surprisingly, it achieves competitive or state-of-the-art performance on parts of the D4RL benchmark, suggesting iterative methods may not always be necessary.
The goal of offline reinforcement learning (RL) is to find an optimal policy given prerecorded trajectories. Many current approaches customize existing off-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated. However, the convergence of such approaches is not guaranteed due to the use of complex non-linear function approximation and an intertwined optimization process. By contrast, we propose a simple baseline algorithm for offline RL that only performs the policy evaluation step once so that the algorithm does not require complex stabilization schemes. Since the proposed algorithm is not likely to converge to an optimal policy, it is an appropriate baseline for actor-critic algorithms that ought to be outperformed if there is indeed value in iterative optimization in the offline setting. Surprisingly, we empirically find that the proposed algorithm exhibits competitive and sometimes even state-of-the-art performance in a subset of the D4RL offline RL benchmark. This result suggests that future work is needed to fully exploit the potential advantages of iterative optimization in order to justify the reduced stability of such methods.