Scaling LLaNA: Advancing NeRF-Language Understanding Through Large-Scale Training
It addresses the limitation of existing MLLMs in comprehensively representing object geometry and appearance, offering a novel approach for NeRF-language understanding, though it is incremental in applying MLLMs to a new data type.
This work tackles the problem of enabling Multimodal Large Language Models (MLLMs) to understand Neural Radiance Fields (NeRFs) by introducing LLaNA, the first MLLM that directly processes NeRF weights for tasks like captioning and Q&A, achieving better performance than methods using derived 2D or 3D representations.
Recent advances in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities in understanding both images and 3D data, yet these modalities face inherent limitations in comprehensively representing object geometry and appearance. Neural Radiance Fields (NeRFs) have emerged as a promising alternative, encoding both geometric and photorealistic properties within the weights of a simple Multi-Layer Perceptron (MLP). This work investigates the feasibility and effectiveness of ingesting NeRFs into an MLLM. We introduce LLaNA, the first MLLM able to perform new tasks such as NeRF captioning and Q\&A, by directly processing the weights of a NeRF's MLP. Notably, LLaNA is able to extract information about the represented objects without the need to render images or materialize 3D data structures. In addition, we build the first large-scale NeRF-language dataset, composed by more than 300K NeRFs trained on ShapeNet and Objaverse, with paired textual annotations that enable various NeRF-language tasks. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that directly processing NeRF weights leads to better performance on NeRF-Language tasks compared to approaches that rely on either 2D or 3D representations derived from NeRFs.