Anti-Exploration by Random Network Distillation
This addresses a limitation in offline RL for improving uncertainty estimation without ensembles, though it is incremental as it builds on existing RND and FiLM techniques.
The paper tackled the problem of Random Network Distillation (RND) being ineffective as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning, and showed that using Feature-wise Linear Modulation (FiLM) conditioning enables RND to achieve performance comparable to ensemble-based methods on the D4RL benchmark.
Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.