DaReNeRF: Direction-aware Representation for Dynamic Scenes
This work addresses the problem of high-fidelity rendering for complex dynamic scenes in computer vision, representing an incremental improvement over existing plane-based methods.
The paper tackles the challenge of modeling and re-rendering dynamic scenes by introducing DaReNeRF, a direction-aware representation that captures scene dynamics from six directions and uses wavelet transformations to recover plane-based information, achieving state-of-the-art performance in novel view synthesis with a 2x reduction in training time compared to prior methods.
Addressing the intricate challenge of modeling and re-rendering dynamic scenes, most recent approaches have sought to simplify these complexities using plane-based explicit representations, overcoming the slow training time issues associated with methods like Neural Radiance Fields (NeRF) and implicit representations. However, the straightforward decomposition of 4D dynamic scenes into multiple 2D plane-based representations proves insufficient for re-rendering high-fidelity scenes with complex motions. In response, we present a novel direction-aware representation (DaRe) approach that captures scene dynamics from six different directions. This learned representation undergoes an inverse dual-tree complex wavelet transformation (DTCWT) to recover plane-based information. DaReNeRF computes features for each space-time point by fusing vectors from these recovered planes. Combining DaReNeRF with a tiny MLP for color regression and leveraging volume rendering in training yield state-of-the-art performance in novel view synthesis for complex dynamic scenes. Notably, to address redundancy introduced by the six real and six imaginary direction-aware wavelet coefficients, we introduce a trainable masking approach, mitigating storage issues without significant performance decline. Moreover, DaReNeRF maintains a 2x reduction in training time compared to prior art while delivering superior performance.