Improving Robot Dual-System Motor Learning with Intrinsically Motivated Meta-Control and Latent-Space Experience Imagination
This addresses performance degradation in robotic tasks like grasping, offering improved sample efficiency, but it is incremental as it builds on existing dual-system approaches.
The paper tackles the problem of compounding prediction errors in dual-system robot motor learning by introducing a meta-controller that arbitrates between model-based and model-free decisions based on local model reliability, integrated with latent-space imagination for training. The results show it outperforms baseline and state-of-the-art methods, learning near-optimal grasping policies in both dense- and sparse-reward environments.
Combining model-based and model-free learning systems has been shown to improve the sample efficiency of learning to perform complex robotic tasks. However, dual-system approaches fail to consider the reliability of the learned model when it is applied to make multiple-step predictions, resulting in a compounding of prediction errors and performance degradation. In this paper, we present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions based on an estimate of the local reliability of the learned model. The reliability estimate is used in computing an intrinsic feedback signal, encouraging actions that lead to data that improves the model. Our approach also integrates arbitration with imagination where a learned latent-space model generates imagined experiences, based on its local reliability, to be used as additional training data. We evaluate our approach against baseline and state-of-the-art methods on learning vision-based robotic grasping in simulation and real world. The results show that our approach outperforms the compared methods and learns near-optimal grasping policies in dense- and sparse-reward environments.