Fast SceneScript: Accurate and Efficient Structured Language Model via Multi-Token Prediction
This addresses efficiency bottlenecks in perception-generalist models for 3D scene understanding, offering a domain-specific incremental improvement.
The paper tackles the slow inference of autoregressive language models for 3D scene layout estimation by introducing Fast SceneScript, which uses multi-token prediction to accelerate inference by generating up to 9 tokens per step without accuracy loss, adding only ~7.5% parameters.
Recent perception-generalist approaches based on language models have achieved state-of-the-art results across diverse tasks, including 3D scene layout estimation, via unified architecture and interface. However, these approaches rely on autoregressive next-token prediction, which is inherently slow. In this work, we introduce Fast SceneScript, a novel structured language model for accurate and efficient 3D scene layout estimation. Our method employs multi-token prediction (MTP) to reduce the number of autoregressive iterations and significantly accelerate inference. While MTP improves speed, unreliable token predictions can significantly reduce accuracy. To filter out unreliable tokens, we adapt self-speculative decoding (SSD) for structured language models and introduce confidence-guided decoding (CGD) with an improved scoring mechanism for token reliability. Furthermore, we design a parameter-efficient mechanism that reduces the parameter overhead of MTP. Extensive experiments on the ASE and Structured3D benchmarks demonstrate that Fast SceneScript can generate up to 9 tokens per decoder inference step without compromising accuracy, while adding only $\sim7.5\%$ additional parameters.