ILabel: Interactive Neural Scene Labelling
This addresses the need for efficient, data-free semantic labeling in 3D scene reconstruction, particularly for real-time applications, though it is incremental as it builds on existing neural field techniques.
The paper tackles the problem of dense semantic labeling of 3D scenes by introducing iLabel, a system that uses a neural field to enable accurate labeling from ultra-sparse user interactions in real-time, achieving higher accuracy than standard pre-trained methods with only tens of clicks.
Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. ILabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.