Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuits
For researchers in quantum machine learning and reinforcement learning, this work provides design principles for parameter-efficient hybrid agents, though the quantum option-value estimation bottleneck limits performance.
This work develops a hybrid hierarchical reinforcement learning agent using variational quantum circuits within the option-critic architecture, achieving up to 66% reduction in trainable parameters while outperforming classical baselines on standard benchmarks.
Reinforcement learning is one of the most challenging learning paradigms where efficacy and efficiency gains are extremely valuable. Hierarchical reinforcement learning is a variant that leverages temporal abstraction to structure decision-making. While parametrized quantum computations have shown success in non-hierarchical reinforcement learning, whether these advantages adapt to hierarchical decision-making remains a critical open question. In this work, we develop a hybrid hierarchical agent based on the option-critic architecture. This hybrid agent substitutes classical components with variational quantum circuits for feature extractors, option-value functions, termination functions, and intra-option policies. Evaluated on standard benchmarking environments, results show that a hybrid agent utilizing a quantum feature extractor can outperform classical baselines while saving up to 66\% trainable parameters. We also identify an architectural bottleneck that quantum option-value estimation severely degrades performance. Further ablation studies reveal how architectural choices of the quantum circuits affect performance. Our work establishes design principles for parameter-efficient hybrid hierarchical agents.