MIMEx: Intrinsic Rewards from Masked Input Modeling
This work addresses exploration challenges in reinforcement learning for robotics and AI, offering a flexible method to improve performance in sparse-reward settings, though it is incremental as it builds on existing conditional prediction approaches.
The paper tackled the problem of exploration in high-dimensional environments by proposing MIMEx, a framework for deriving intrinsic rewards from masked input modeling, and demonstrated superior results on challenging sparse-reward visuomotor tasks compared to baselines.
Exploring in environments with high-dimensional observations is hard. One promising approach for exploration is to use intrinsic rewards, which often boils down to estimating "novelty" of states, transitions, or trajectories with deep networks. Prior works have shown that conditional prediction objectives such as masked autoencoding can be seen as stochastic estimation of pseudo-likelihood. We show how this perspective naturally leads to a unified view on existing intrinsic reward approaches: they are special cases of conditional prediction, where the estimation of novelty can be seen as pseudo-likelihood estimation with different mask distributions. From this view, we propose a general framework for deriving intrinsic rewards -- Masked Input Modeling for Exploration (MIMEx) -- where the mask distribution can be flexibly tuned to control the difficulty of the underlying conditional prediction task. We demonstrate that MIMEx can achieve superior results when compared against competitive baselines on a suite of challenging sparse-reward visuomotor tasks.