Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generation
This addresses a bottleneck for computer vision applications by enabling efficient and accurate simulation of camera effects, though it is incremental as it builds on existing methods like 4D Gaussian Splatting and ray tracing.
The paper tackles the problem of generating training data with realistic camera effects like fisheye distortion and rolling shutter, which are lacking in computer vision systems, by proposing 4D Gaussian Ray Tracing (4D-GRT) that achieves the fastest rendering speed with comparable or better quality than baselines.
Common computer vision systems typically assume ideal pinhole cameras but fail when facing real-world camera effects such as fisheye distortion and rolling shutter, mainly due to the lack of learning from training data with camera effects. Existing data generation approaches suffer from either high costs, sim-to-real gaps or fail to accurately model camera effects. To address this bottleneck, we propose 4D Gaussian Ray Tracing (4D-GRT), a novel two-stage pipeline that combines 4D Gaussian Splatting with physically-based ray tracing for camera effect simulation. Given multi-view videos, 4D-GRT first reconstructs dynamic scenes, then applies ray tracing to generate videos with controllable, physically accurate camera effects. 4D-GRT achieves the fastest rendering speed while performing better or comparable rendering quality compared to existing baselines. Additionally, we construct eight synthetic dynamic scenes in indoor environments across four camera effects as a benchmark to evaluate generated videos with camera effects.