Sphinx: Efficiently Serving Novel View Synthesis using Regression-Guided Selective Refinement
This work addresses the problem of computational inefficiency in novel view synthesis for applications requiring real-time or low-latency image generation, offering a significant improvement in speed without sacrificing quality.
The paper tackles the challenge of achieving high-quality novel view synthesis with efficient inference by presenting Sphinx, a training-free hybrid framework that uses regression-based initialization and selective refinement to reduce computational load, resulting in a 1.8x speedup over diffusion models with less than 5% perceptual degradation.
Novel View Synthesis (NVS) is the task of generating new images of a scene from viewpoints that were not part of the original input. Diffusion-based NVS can generate high-quality, temporally consistent images, however, remains computationally prohibitive. Conversely, regression-based NVS offers suboptimal generation quality despite requiring significantly lower compute; leaving the design objective of a high-quality, inference-efficient NVS framework an open challenge. To close this critical gap, we present Sphinx, a training-free hybrid inference framework that achieves diffusion-level fidelity at a significantly lower compute. Sphinx proposes to use regression-based fast initialization to guide and reduce the denoising workload for the diffusion model. Additionally, it integrates selective refinement with adaptive noise scheduling, allowing more compute to uncertain regions and frames. This enables Sphinx to provide flexible navigation of the performance-quality trade-off, allowing adaptation to latency and fidelity requirements for dynamically changing inference scenarios. Our evaluation shows that Sphinx achieves an average 1.8x speedup over diffusion model inference with negligible perceptual degradation of less than 5%, establishing a new Pareto frontier between quality and latency in NVS serving.