Beyond Point-Attached Semantics: Object-Centric Semantic Fields for Generalizable Manipulation
For robot manipulation, this work addresses the instability of point-attached semantics across viewpoints and instances, offering a more robust object-level representation.
The paper proposes an object-centric continuous semantic field that provides stable part-aware embeddings at arbitrary 3D locations, improving manipulation policy success rates over raw point-cloud and feature-lifting baselines in simulation and real-world bimanual tasks.
Generalizable robot manipulation requires stable 3D understanding of functional object parts, such as handles, tool heads, openings, and graspable regions. Raw point clouds provide geometry but lack explicit part semantics, and their sampled points vary with viewpoint, sensor configuration, and object instance. Existing 2D feature lifting and discrete 3D point-wise features enrich point clouds with semantics, but the resulting features remain attached to observation-dependent samples. We propose an object-centric continuous semantic field that conditions on an object point cloud and reads part-aware semantic embeddings at explicit 3D query locations. The field is trained from part-annotated object models and then frozen to generate semantic point clouds as object-level conditioning for manipulation policies. Experiments on RoboTwin simulation tasks and real-world bimanual object manipulation show that our representation provides more stable functional-part cues and improves policy performance over raw point-cloud, 2D feature lifting, and 3D point-wise feature baselines. Project Page: \href{https://zainzh.github.io/beyond-point-attached-semantics}{https://zainzh.github.io/beyond-point-attached-semantics}.