Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models
This work addresses the problem of efficient 3D semantic understanding from 2D foundation models, offering a lightweight alternative to heavy 3D pretraining for computer vision applications.
Casper3D introduces a lightweight probabilistic framework that converts noisy multi-view 2D foundation-model embeddings into a latent 3D semantic representation, achieving more stable 3D semantics than simple multi-view pooling, especially in ambiguous and noisy settings.
We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic representation. We model view-level semantic features as noisy observations of an underlying 3D semantic state and infer this state with a set-based variational model that incorporates relative pose during multi-view reasoning. Casper3D is trained by predicting held-out semantic observations from novel viewpoints, while remaining aligned with visual and text semantic spaces for open-vocabulary 3D understanding. The framework is backbone-agnostic and applies to both language-aligned and self-supervised embeddings. Experiments show that Casper3D produces more stable 3D semantics than simple multi-view pooling, especially in ambiguous and noisy settings.