Video prediction using score-based conditional density estimation
This work addresses the challenge of high-dimensional probabilistic inference in video prediction for natural scenes, offering a method that improves handling of ambiguity, though it appears incremental in its approach to existing deep learning techniques.
The paper tackles the problem of representing uncertainty in video frame prediction by developing a score-based conditional density estimation framework, which learns to sample likely future frames from previous ones and handles occlusion boundaries by selecting among probable trajectories rather than averaging.
Temporal prediction is inherently uncertain, but representing the ambiguity in natural image sequences is a challenging high-dimensional probabilistic inference problem. For natural scenes, the curse of dimensionality renders explicit density estimation statistically and computationally intractable. Here, we describe an implicit regression-based framework for learning and sampling the conditional density of the next frame in a video given previous observed frames. We show that sequence-to-image deep networks trained on a simple resilience-to-noise objective function extract adaptive representations for temporal prediction. Synthetic experiments demonstrate that this score-based framework can handle occlusion boundaries: unlike classical methods that average over bifurcating temporal trajectories, it chooses among likely trajectories, selecting more probable options with higher frequency. Furthermore, analysis of networks trained on natural image sequences reveals that the representation automatically weights predictive evidence by its reliability, which is a hallmark of statistical inference