Quantum enhancements for deep reinforcement learning in large spaces
This work addresses the problem of inefficient deep reinforcement learning in large spaces for researchers and practitioners in quantum machine learning, offering incremental advancements by integrating quantum methods to improve classical sampling bottlenecks.
The paper tackles the challenge of applying quantum enhancements to deep reinforcement learning, which has seen limited quantum improvements due to a lack of computational bottlenecks, and demonstrates that models with sampling bottlenecks, such as energy-based models, provide substantial learning advantages in complex environments, with proposed quantum algorithms to alleviate computational costs.
In the past decade, the field of quantum machine learning has drawn significant attention due to the prospect of bringing genuine computational advantages to now widespread algorithmic methods. However, not all domains of machine learning have benefited equally from quantum enhancements. Notably, deep learning and reinforcement learning, despite their tremendous success in the classical domain, both individually and combined, remain relatively unaddressed by the quantum community. Arguably, one reason behind this is the systematic use in these domains of models and methods without prominent computational bottlenecks, leaving little room for quantum improvements. In this work, we study the state-of-the-art neural-network approaches for reinforcement learning with quantum enhancements in mind. We demonstrate the substantial learning advantage that models with a sampling bottleneck can provide over conventional neural network architectures in complex learning environments. These so-called energy-based models, like deep energy-based reinforcement learning, and deep projective simulation that we also introduce in this work, effectively allow to trade off learning performance for efficiency of computation. To alleviate the additional computational costs, we propose to leverage future and near-term quantum algorithms, resulting in overall more advantageous learning algorithms. This is achieved using cutting-edge and new quantum computing machinery to speed-up classical sampling methods and by employing generalized models to gain an additional quantum advantage.