Learning Control by Iterative Inversion
This addresses the challenge of imitation learning in robotics or AI control without requiring explicit rewards or action labels, though it is incremental as it builds on existing supervised learning and embedding techniques.
The paper tackles the problem of learning control without input-output pairs or rewards by proposing iterative inversion, which learns an inverse function using only samples from the desired output distribution and access to the forward function. The result demonstrates non-trivial continuous control on several tasks and improved performance in imitating diverse behaviors compared to reward-based methods.
We propose $\textit{iterative inversion}$ -- an algorithm for learning an inverse function without input-output pairs, but only with samples from the desired output distribution and access to the forward function. The key challenge is a $\textit{distribution shift}$ between the desired outputs and the outputs of an initial random guess, and we prove that iterative inversion can steer the learning correctly, under rather strict conditions on the function. We apply iterative inversion to learn control. Our input is a set of demonstrations of desired behavior, given as video embeddings of trajectories (without actions), and our method iteratively learns to imitate trajectories generated by the current policy, perturbed by random exploration noise. Our approach does not require rewards, and only employs supervised learning, which can be easily scaled to use state-of-the-art trajectory embedding techniques and policy representations. Indeed, with a VQ-VAE embedding, and a transformer-based policy, we demonstrate non-trivial continuous control on several tasks. Further, we report an improved performance on imitating diverse behaviors compared to reward based methods.