Vector Quantized Feature Fields for Fast 3D Semantic Lifting
This addresses the storage and query efficiency challenges in 3D scene representation for applications in computer vision and robotics, though it appears incremental as it builds on existing feature field and lifting techniques.
The paper tackles the problem of efficiently retrieving pixel-aligned relevance masks for 3D semantic lifting by introducing Vector-Quantized Feature Fields, which enable lightweight on-demand retrieval. The method demonstrates effectiveness in complex scenes and enables applications like text-driven scene editing and improved embodied question answering efficiency.
We generalize lifting to semantic lifting by incorporating per-view masks that indicate relevant pixels for lifting tasks. These masks are determined by querying corresponding multiscale pixel-aligned feature maps, which are derived from scene representations such as distilled feature fields and feature point clouds. However, storing per-view feature maps rendered from distilled feature fields is impractical, and feature point clouds are expensive to store and query. To enable lightweight on-demand retrieval of pixel-aligned relevance masks, we introduce the Vector-Quantized Feature Field. We demonstrate the effectiveness of the Vector-Quantized Feature Field on complex indoor and outdoor scenes. Semantic lifting, when paired with a Vector-Quantized Feature Field, can unlock a myriad of applications in scene representation and embodied intelligence. Specifically, we showcase how our method enables text-driven localized scene editing and significantly improves the efficiency of embodied question answering.