Combining Model-Free Q-Ensembles and Model-Based Approaches for Informed Exploration
This work addresses exploration challenges in reinforcement learning, but it appears incremental as it combines existing methods without introducing a fundamentally new paradigm.
The paper tackled the problem of improving exploration in reinforcement learning by integrating model-free Q-ensembles with model-based approaches, resulting in superior performance compared to using Q-ensembles alone.
Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, model-based approaches, such as encoder-decoder models have been used successfully for next frame prediction given previous frames. This paper proposes to integrate the model-free Q-ensembles and model-based approaches with the hope of compounding the benefits of both and achieving superior exploration as a result. Results show that a model-based trajectory memory approach when combined with Q-ensembles produces superior performance when compared to only using Q-ensembles.