TextSplat: Text-Guided Semantic Fusion for Generalizable Gaussian Splatting
This work addresses the need for more accurate fine-grained 3D reconstruction in complex scenes, which is important for applications in computer vision and robotics, though it appears incremental by building on existing generalizable Gaussian splatting methods.
The paper tackles the problem of enhancing semantic understanding in generalizable Gaussian splatting for 3D reconstruction by introducing TextSplat, a text-driven framework that integrates semantic cues to improve alignment between geometry and semantics, resulting in high-fidelity reconstructions with improved performance on benchmark datasets.
Recent advancements in Generalizable Gaussian Splatting have enabled robust 3D reconstruction from sparse input views by utilizing feed-forward Gaussian Splatting models, achieving superior cross-scene generalization. However, while many methods focus on geometric consistency, they often neglect the potential of text-driven guidance to enhance semantic understanding, which is crucial for accurately reconstructing fine-grained details in complex scenes. To address this limitation, we propose TextSplat--the first text-driven Generalizable Gaussian Splatting framework. By employing a text-guided fusion of diverse semantic cues, our framework learns robust cross-modal feature representations that improve the alignment of geometric and semantic information, producing high-fidelity 3D reconstructions. Specifically, our framework employs three parallel modules to obtain complementary representations: the Diffusion Prior Depth Estimator for accurate depth information, the Semantic Aware Segmentation Network for detailed semantic information, and the Multi-View Interaction Network for refined cross-view features. Then, in the Text-Guided Semantic Fusion Module, these representations are integrated via the text-guided and attention-based feature aggregation mechanism, resulting in enhanced 3D Gaussian parameters enriched with detailed semantic cues. Experimental results on various benchmark datasets demonstrate improved performance compared to existing methods across multiple evaluation metrics, validating the effectiveness of our framework. The code will be publicly available.