LAOF: Robust Latent Action Learning with Optical Flow Constraints
This addresses the challenge of scalable embodied foundation model pre-training by reducing reliance on scarce action labels, though it is incremental as it builds on existing latent action learning methods.
The paper tackles the problem of learning latent actions from videos with action-irrelevant distractors by proposing LAOF, a pseudo-supervised framework that uses optical flow constraints to improve robustness; it shows superior performance on downstream tasks, matching or surpassing action-supervised methods even without supervision.
Learning latent actions from large-scale videos is crucial for the pre-training of scalable embodied foundation models, yet existing methods often struggle with action-irrelevant distractors. Although incorporating action supervision can alleviate these distractions, its effectiveness is restricted by the scarcity of available action labels. Optical flow represents pixel-level motion between consecutive frames, naturally suppressing background elements and emphasizing moving objects. Motivated by this, we propose robust Latent Action learning with Optical Flow constraints, called LAOF, a pseudo-supervised framework that leverages the agent's optical flow as an action-driven signal to learn latent action representations robust to distractors. Experimental results show that the latent representations learned by LAOF outperform existing methods on downstream imitation learning and reinforcement learning tasks. This superior performance arises from optical flow constraints, which substantially stabilize training and improve the quality of latent representations under extremely label-scarce conditions, while remaining effective as the proportion of action labels increases to 10 percent. Importantly, even without action supervision, LAOF matches or surpasses action-supervised methods trained with 1 percent of action labels.