LERF: Language Embedded Radiance Fields
This work addresses the challenge of interpreting and interacting with 3D environments using natural language, with potential applications in robotics and vision-language understanding, representing a novel integration rather than an incremental improvement.
The paper tackles the problem of enabling open-ended language queries in 3D scenes by proposing Language Embedded Radiance Fields (LERF), which grounds CLIP language embeddings into NeRF to extract 3D relevancy maps for a broad range of prompts interactively in real-time.
Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. The project website can be found at https://lerf.io .