A Temporally Correlated Latent Exploration for Reinforcement Learning
This work addresses exploration challenges in reinforcement learning for AI agents, offering a method that is robust to noisy and stochastic environments, though it appears incremental as it builds on existing intrinsic reward approaches.
The paper tackled the problem of efficient exploration in deep reinforcement learning by proposing TeCLE, a novel intrinsic reward formulation that uses an action-conditioned latent space and temporal correlation to address vulnerabilities to Noisy TV and stochasticity, achieving robustness in benchmark environments like Minigrid and Stochastic Atari.
Efficient exploration remains one of the longstanding problems of deep reinforcement learning. Instead of depending solely on extrinsic rewards from the environments, existing methods use intrinsic rewards to enhance exploration. However, we demonstrate that these methods are vulnerable to Noisy TV and stochasticity. To tackle this problem, we propose Temporally Correlated Latent Exploration (TeCLE), which is a novel intrinsic reward formulation that employs an action-conditioned latent space and temporal correlation. The action-conditioned latent space estimates the probability distribution of states, thereby avoiding the assignment of excessive intrinsic rewards to unpredictable states and effectively addressing both problems. Whereas previous works inject temporal correlation for action selection, the proposed method injects it for intrinsic reward computation. We find that the injected temporal correlation determines the exploratory behaviors of agents. Various experiments show that the environment where the agent performs well depends on the amount of temporal correlation. To the best of our knowledge, the proposed TeCLE is the first approach to consider the action conditioned latent space and temporal correlation for curiosity-driven exploration. We prove that the proposed TeCLE can be robust to the Noisy TV and stochasticity in benchmark environments, including Minigrid and Stochastic Atari.