StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects
This work addresses the challenge for assistive robots in warehouses, offices, and homes to handle unseen objects and complex arrangements, representing an incremental advance over prior methods focused on simpler pairwise relations.
The paper tackles the problem of enabling robots to rearrange novel objects into semantically meaningful structures based on language commands, and demonstrates that their proposed StructFormer model allows a physical robot to successfully perform such rearrangements with multi-object relational constraints.
Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without significant programming. While previous works have examined recognizing pairwise semantic relations and sequential manipulation to change these simple relations none have shown the ability to arrange objects into complex structures such as circles or table settings. To address this problem we propose a novel transformer-based neural network, StructFormer, which takes as input a partial-view point cloud of the current object arrangement and a structured language command encoding the desired object configuration. We show through rigorous experiments that StructFormer enables a physical robot to rearrange novel objects into semantically meaningful structures with multi-object relational constraints inferred from the language command.