Neural Light Spheres for Implicit Image Stitching and View Synthesis
This addresses the problem of creating robust panoramas for mobile camera users, though it is an incremental improvement over existing neural rendering techniques.
The paper tackles the challenge of creating high-quality panoramas from mobile video captures by introducing a spherical neural light field model that handles depth parallax, view-dependent lighting, and scene motion. It achieves real-time rendering at 50 FPS with 80 MB per scene and shows improved quality over traditional methods.
Challenging to capture, and challenging to display on a cellphone screen, the panorama paradoxically remains both a staple and underused feature of modern mobile camera applications. In this work we address both of these challenges with a spherical neural light field model for implicit panoramic image stitching and re-rendering; able to accommodate for depth parallax, view-dependent lighting, and local scene motion and color changes during capture. Fit during test-time to an arbitrary path panoramic video capture -- vertical, horizontal, random-walk -- these neural light spheres jointly estimate the camera path and a high-resolution scene reconstruction to produce novel wide field-of-view projections of the environment. Our single-layer model avoids expensive volumetric sampling, and decomposes the scene into compact view-dependent ray offset and color components, with a total model size of 80 MB per scene, and real-time (50 FPS) rendering at 1080p resolution. We demonstrate improved reconstruction quality over traditional image stitching and radiance field methods, with significantly higher tolerance to scene motion and non-ideal capture settings.