Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions
This addresses the challenge of sequential decision-making in real-world settings with parameterized actions, offering a novel approach for long-horizon, sparse-reward tasks.
The paper tackles the problem of reinforcement learning in parameterized action spaces, which combine discrete actions with continuous parameters, by enabling agents to autonomously learn state and action abstractions online, resulting in TD(λ) achieving markedly higher sample efficiency than state-of-the-art baselines across several domains.
Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD($λ$) to achieve markedly higher sample efficiency than state-of-the-art baselines.