Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting
This work addresses the challenge of data redundancy in semantic and dynamic scene understanding for embodied agents, offering a principled alternative to heuristic strategies.
The paper tackles the problem of selecting informative views for training models in semantic and dynamic 3D Gaussian splatting by formulating it as an active learning problem, resulting in improved rendering quality and semantic segmentation performance over baseline methods.
Understanding semantics and dynamics has been crucial for embodied agents in various tasks. Both tasks have much more data redundancy than the static scene understanding task. We formulate the view selection problem as an active learning problem, where the goal is to prioritize frames that provide the greatest information gain for model training. To this end, we propose an active learning algorithm with Fisher Information that quantifies the informativeness of candidate views with respect to both semantic Gaussian parameters and deformation networks. This formulation allows our method to jointly handle semantic reasoning and dynamic scene modeling, providing a principled alternative to heuristic or random strategies. We evaluate our method on large-scale static images and dynamic video datasets by selecting informative frames from multi-camera setups. Experimental results demonstrate that our approach consistently improves rendering quality and semantic segmentation performance, outperforming baseline methods based on random selection and uncertainty-based heuristics.