Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping
This work addresses risk-aware robotic grasping for robotics applications, but it is incremental as it adapts existing methods to a new domain.
The authors tackled the challenge of applying distributional reinforcement learning from discrete settings to complex, noisy, and continuous robotic grasping tasks by proposing Quantile QT-Opt (Q2-Opt), which achieved a superior vision-based object grasping success rate and improved sample efficiency. They also found that previous batch RL findings from arcade games did not generalize to their setup.
The distributional perspective on reinforcement learning (RL) has given rise to a series of successful Q-learning algorithms, resulting in state-of-the-art performance in arcade game environments. However, it has not yet been analyzed how these findings from a discrete setting translate to complex practical applications characterized by noisy, high dimensional and continuous state-action spaces. In this work, we propose Quantile QT-Opt (Q2-Opt), a distributional variant of the recently introduced distributed Q-learning algorithm for continuous domains, and examine its behaviour in a series of simulated and real vision-based robotic grasping tasks. The absence of an actor in Q2-Opt allows us to directly draw a parallel to the previous discrete experiments in the literature without the additional complexities induced by an actor-critic architecture. We demonstrate that Q2-Opt achieves a superior vision-based object grasping success rate, while also being more sample efficient. The distributional formulation also allows us to experiment with various risk distortion metrics that give us an indication of how robots can concretely manage risk in practice using a Deep RL control policy. As an additional contribution, we perform batch RL experiments in our virtual environment and compare them with the latest findings from discrete settings. Surprisingly, we find that the previous batch RL findings from the literature obtained on arcade game environments do not generalise to our setup.