FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes
This addresses the challenge of data scarcity for off-trajectory views in driving simulations, which is incremental as it builds on existing methods with a new pipeline.
The paper tackles the problem of high-quality camera simulation for autonomous driving from viewpoints not recorded in training data, achieving synthesis under large deviations of over 3 meters.
We propose FreeSim, a camera simulation method for autonomous driving. FreeSim emphasizes high-quality rendering from viewpoints beyond the recorded ego trajectories. In such viewpoints, previous methods have unacceptable degradation because the training data of these viewpoints is unavailable. To address such data scarcity, we first propose a generative enhancement model with a matched data construction strategy. The resulting model can generate high-quality images in a viewpoint slightly deviated from the recorded trajectories, conditioned on the degraded rendering of this viewpoint. We then propose a progressive reconstruction strategy, which progressively adds generated images of unrecorded views into the reconstruction process, starting from slightly off-trajectory viewpoints and moving progressively farther away. With this progressive generation-reconstruction pipeline, FreeSim supports high-quality off-trajectory view synthesis under large deviations of more than 3 meters.