Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning
For autonomous robots, this work addresses the problem of abstracting continuous sensorimotor interactions into discrete representations for manipulation planning, offering improved planning precision over existing methods.
The paper proposes a model that jointly discovers manipulation primitives and object categories from random interaction data by predicting multi-modal outcomes, enabling effect-driven planning that outperforms state-of-the-art and visual-based methods in tabletop repositioning and stacking tasks.
To perform complex manipulation planning, autonomous robots are required to abstract continuous, high-dimensional sensorimotor interactions into discrete object and action representations. Earlier work either categorized objects based on visual appearances, which fails to distinguish objects that appear similar but behave differently, or based on effects under interaction, but was limited to predefined actions. To address these limitations, we propose a model that jointly discovers high-level manipulation primitives and object categories through a binary bottleneck layer, trained to predict multi-modal outcomes, including object motion, contact, and force feedback, from random interaction data. Building on these discovered binary representations, we leverage a discrete planning method that uses intermediate steps in the predicted effect trajectory to enable partial action executions for precise low-level control. Additionally, we evaluate our framework's generalization capabilities on novel objects by assigning object categories through comparing a small number of interaction effects with the predicted effects of learned object symbols, enabling few-shot generalization based on behavior rather than visual similarity. We conduct experiments on tabletop repositioning and stacking tasks, and confirm that our effect-driven planning approach outperforms both a state-of-the-art method and a visual-based alternative in planning precision across seen and novel objects.