Hard-Thresholding Meets Evolution Strategies in Reinforcement Learning
This addresses a limitation in Evolution Strategies for real-world RL applications where irrelevant features are common, representing an incremental improvement.
The paper tackles the problem of irrelevant features in Evolution Strategies for reinforcement learning by proposing NESHT, which integrates Hard-Thresholding to promote sparsity, resulting in improved performance in noisy Mujoco and Atari tasks.
Evolution Strategies (ES) have emerged as a competitive alternative for model-free reinforcement learning, showcasing exemplary performance in tasks like Mujoco and Atari. Notably, they shine in scenarios with imperfect reward functions, making them invaluable for real-world applications where dense reward signals may be elusive. Yet, an inherent assumption in ES, that all input features are task-relevant, poses challenges, especially when confronted with irrelevant features common in real-world problems. This work scrutinizes this limitation, particularly focusing on the Natural Evolution Strategies (NES) variant. We propose NESHT, a novel approach that integrates Hard-Thresholding (HT) with NES to champion sparsity, ensuring only pertinent features are employed. Backed by rigorous analysis and empirical tests, NESHT demonstrates its promise in mitigating the pitfalls of irrelevant features and shines in complex decision-making problems like noisy Mujoco and Atari tasks.