From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback
For imitation learning practitioners, this work addresses the practical problem of learning from imperfect human feedback, offering a more robust alternative to standard behavior cloning.
Behavior cloning is brittle with imperfect human action labels. The authors propose CLIC, which uses set-valued action targets from human corrections instead of pointwise labels, achieving competitive performance with accurate data and substantially more robustness under noisy, relative, and partial feedback.
Behavior cloning (BC) optimizes policies by treating human demonstrations as pointwise action labels. While effective with accurate action labels, this formulation is brittle in practice: when human-provided actions are imperfect, treating each label as an exact target can steer the policy away from the underlying desired behavior, particularly when expressive models are used (e.g., energy-based models). As a result, we propose a human-in-the-loop alternative that replaces pointwise supervision with set-valued action targets. We introduce Contrastive policy Learning from Interactive Corrections (CLIC). CLIC leverages human corrections to construct and refine sets of desired actions, and optimizes a policy to place probability mass over these sets rather than over a single action target. This formulation naturally accommodates both absolute and relative corrections and can represent complex multi-modal behaviors. Extensive simulation and real-robot experiments show that the proposed approach leads to effective policy learning across diverse settings: CLIC remains competitive with the state of the art under accurate data while being substantially more robust under noisy, relative, and partial feedback. Our implementation is publicly available at https://clic-webpage.github.io/.