Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
This work addresses sample efficiency in reinforcement learning, offering an incremental improvement over conventional entropy regularization methods.
The paper tackled the problem of sample-efficient exploration in reinforcement learning by proposing sample-aware entropy regularization, which leverages replay buffer samples to enhance policy entropy. The resulting Diversity Actor-Critic (DAC) algorithm significantly outperformed existing recent RL algorithms in numerical results.
In this paper, sample-aware policy entropy regularization is proposed to enhance the conventional policy entropy regularization for better exploration. Exploiting the sample distribution obtainable from the replay buffer, the proposed sample-aware entropy regularization maximizes the entropy of the weighted sum of the policy action distribution and the sample action distribution from the replay buffer for sample-efficient exploration. A practical algorithm named diversity actor-critic (DAC) is developed by applying policy iteration to the objective function with the proposed sample-aware entropy regularization. Numerical results show that DAC significantly outperforms existing recent algorithms for reinforcement learning.