SceneTok: A Compressed, Diffusable Token Space for 3D Scenes
This addresses the need for efficient 3D scene representation and generation for applications in computer vision and graphics, offering a novel approach with significant compression and speed improvements.
The paper tackles the problem of representing and generating 3D scenes by introducing SceneTok, a tokenizer that encodes scenes into compressed, diffusable tokens, achieving 1-3 orders of magnitude stronger compression than other representations while reaching state-of-the-art reconstruction quality and enabling scene generation in 5 seconds.
We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.