PerLA: Perceptive 3D Language Assistant
This work addresses the problem of 3D scene understanding for AI systems, offering a perceptive assistant that improves performance in tasks like question answering and dense captioning, though it appears incremental by building on existing 3D language assistant frameworks.
The paper tackles the challenge of enabling Large Language Models to understand 3D physical worlds by addressing the loss of details or context in current point cloud processing methods, resulting in PerLA, which outperforms state-of-the-art 3D language assistants with gains such as +1.34 CiDEr on ScanQA for question answering.
Enabling Large Language Models (LLMs) to understand the 3D physical world is an emerging yet challenging research direction. Current strategies for processing point clouds typically downsample the scene or divide it into smaller parts for separate analysis. However, both approaches risk losing key local details or global contextual information. In this paper, we introduce PerLA, a 3D language assistant designed to be more perceptive to both details and context, making visual representations more informative for the LLM. PerLA captures high-resolution (local) details in parallel from different point cloud areas and integrates them with (global) context obtained from a lower-resolution whole point cloud. We present a novel algorithm that preserves point cloud locality through the Hilbert curve and effectively aggregates local-to-global information via cross-attention and a graph neural network. Lastly, we introduce a novel loss for local representation consensus to promote training stability. PerLA outperforms state-of-the-art 3D language assistants, with gains of up to +1.34 CiDEr on ScanQA for question answering, and +4.22 on ScanRefer and +3.88 on Nr3D for dense captioning. https://gfmei.github.io/PerLA/