Learning from Imperfect Demonstrations via Adversarial Confidence Transfer
This addresses the limitation of assuming expert demonstrations in real-world applications where data may be suboptimal or include failures, though it is incremental as it builds on existing learning from demonstration methods.
The paper tackles the problem of learning from imperfect demonstrations by transferring confidence predictors from a source environment with labeled confidence to a target environment with unlabeled demonstrations, using adversarial distribution matching on partial trajectories. Experiments in simulated environments and a real robot task show that this approach learns a policy with the highest expected return.
Existing learning from demonstration algorithms usually assume access to expert demonstrations. However, this assumption is limiting in many real-world applications since the collected demonstrations may be suboptimal or even consist of failure cases. We therefore study the problem of learning from imperfect demonstrations by learning a confidence predictor. Specifically, we rely on demonstrations along with their confidence values from a different correspondent environment (source environment) to learn a confidence predictor for the environment we aim to learn a policy in (target environment -- where we only have unlabeled demonstrations.) We learn a common latent space through adversarial distribution matching of multi-length partial trajectories to enable the transfer of confidence across source and target environments. The learned confidence reweights the demonstrations to enable learning more from informative demonstrations and discarding the irrelevant ones. Our experiments in three simulated environments and a real robot reaching task demonstrate that our approach learns a policy with the highest expected return.