Object-centric Inference for Language Conditioned Placement: A Foundation Model based Approach
This work addresses the problem of enabling robots to interpret and execute complex spatial instructions more efficiently and generally, representing an incremental improvement over previous methods.
The paper tackles the task of language-conditioned object placement for robots by proposing an object-centric framework that uses foundation models to ground reference objects and spatial relations, achieving a 97.75% success rate with only ~0.26M trainable parameters and better generalization to unseen objects and instructions.
We focus on the task of language-conditioned object placement, in which a robot should generate placements that satisfy all the spatial relational constraints in language instructions. Previous works based on rule-based language parsing or scene-centric visual representation have restrictions on the form of instructions and reference objects or require large amounts of training data. We propose an object-centric framework that leverages foundation models to ground the reference objects and spatial relations for placement, which is more sample efficient and generalizable. Experiments indicate that our model can achieve a 97.75% success rate of placement with only ~0.26M trainable parameters. Besides, our method generalizes better to both unseen objects and instructions. Moreover, with only 25% training data, we still outperform the top competing approach.