Material-informed Gaussian Splatting for 3D World Reconstruction in a Digital Twin
This addresses the problem of accurate and efficient 3D world reconstruction for Digital Twins, particularly in applications like sensor simulation, by offering a camera-only alternative to complex LiDAR-camera fusion methods.
The paper tackles 3D reconstruction for Digital Twins by proposing a camera-only pipeline that uses 3D Gaussian Splatting and semantic material masks to achieve sensor simulation fidelity comparable to LiDAR-camera fusion, eliminating hardware complexity and calibration requirements, with validation on an internal dataset using LiDAR as ground truth.
3D reconstruction for Digital Twins often relies on LiDAR-based methods, which provide accurate geometry but lack the semantics and textures naturally captured by cameras. Traditional LiDAR-camera fusion approaches require complex calibration and still struggle with certain materials like glass, which are visible in images but poorly represented in point clouds. We propose a camera-only pipeline that reconstructs scenes using 3D Gaussian Splatting from multi-view images, extracts semantic material masks via vision models, converts Gaussian representations to mesh surfaces with projected material labels, and assigns physics-based material properties for accurate sensor simulation in modern graphics engines and simulators. This approach combines photorealistic reconstruction with physics-based material assignment, providing sensor simulation fidelity comparable to LiDAR-camera fusion while eliminating hardware complexity and calibration requirements. We validate our camera-only method using an internal dataset from an instrumented test vehicle, leveraging LiDAR as ground truth for reflectivity validation alongside image similarity metrics.