LISA-3D: Lifting Language-Image Segmentation to 3D via Multi-View Consistency
This work addresses the challenge of generating consistent 3D content from open-vocabulary language instructions, offering a modular and data-efficient solution for language-guided 3D creation, though it is incremental as it builds on existing models like LISA and SAM-3D.
The paper tackles the problem of text-driven 3D reconstruction by developing LISA-3D, a framework that lifts language-image segmentation to 3D using multi-view consistency, improving language-to-3D accuracy by up to +15.6 points over baselines while adapting only 11.6M parameters.
Text-driven 3D reconstruction demands a mask generator that simultaneously understands open-vocabulary instructions and remains consistent across viewpoints. We present LISA-3D, a two-stage framework that lifts language-image segmentation into 3D by retrofitting the instruction-following model LISA with geometry-aware Low-Rank Adaptation (LoRA) layers and reusing a frozen SAM-3D reconstructor. During training we exploit off-the-shelf RGB-D sequences and their camera poses to build a differentiable reprojection loss that enforces cross-view agreement without requiring any additional 3D-text supervision. The resulting masks are concatenated with RGB images to form RGBA prompts for SAM-3D, which outputs Gaussian splats or textured meshes without retraining. Across ScanRefer and Nr3D, LISA-3D improves language-to-3D accuracy by up to +15.6 points over single-view baselines while adapting only 11.6M parameters. The system is modular, data-efficient, and supports zero-shot deployment on unseen categories, providing a practical recipe for language-guided 3D content creation. Our code will be available at https://github.com/binisalegend/LISA-3D.