Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision
This addresses the challenge of 3D cloth state estimation for robotics or simulation applications, but it appears incremental as it builds on existing methods like Gaussian Splatting.
The paper tackles the problem of estimating 3D cloth states from RGB images by introducing Cloth-Splatting, which uses a prediction-update framework with a dynamics model and 3D Gaussian Splatting, resulting in improved accuracy and reduced convergence time compared to baselines.
We introduce Cloth-Splatting, a method for estimating 3D states of cloth from RGB images through a prediction-update framework. Cloth-Splatting leverages an action-conditioned dynamics model for predicting future states and uses 3D Gaussian Splatting to update the predicted states. Our key insight is that coupling a 3D mesh-based representation with Gaussian Splatting allows us to define a differentiable map between the cloth state space and the image space. This enables the use of gradient-based optimization techniques to refine inaccurate state estimates using only RGB supervision. Our experiments demonstrate that Cloth-Splatting not only improves state estimation accuracy over current baselines but also reduces convergence time.