Beyond Brightness Constancy: Learning Noise Models for Optical Flow
This work addresses a long-standing bottleneck in optical flow estimation for computer vision applications, offering a novel approach to data cost modeling that could enhance accuracy in tasks like motion tracking and video analysis.
The paper tackled the problem of improving optical flow estimation by learning a data cost that models the distribution of noise after warping images, rather than relying on traditional brightness or gradient constancy assumptions. The result showed that a learned Gaussian Mixture Model (GMM) for patch warp errors significantly outperformed existing methods, leading to better optical flow estimation as measured by higher likelihood and improved performance.
Optical flow is typically estimated by minimizing a "data cost" and an optional regularizer. While there has been much work on different regularizers many modern algorithms still use a data cost that is not very different from the ones used over 30 years ago: a robust version of brightness constancy or gradient constancy. In this paper we leverage the recent availability of ground-truth optical flow databases in order to learn a data cost. Specifically we take a generative approach in which the data cost models the distribution of noise after warping an image according to the flow and we measure the "goodness" of a data cost by how well it matches the true distribution of flow warp error. Consistent with current practice, we find that robust versions of gradient constancy are better models than simple brightness constancy but a learned GMM that models the density of patches of warp error gives a much better fit than any existing assumption of constancy. This significant advantage of the GMM is due to an explicit modeling of the spatial structure of warp errors, a feature which is missing from almost all existing data costs in optical flow. Finally, we show how a good density model of warp error patches can be used for optical flow estimation on whole images. We replace the data cost by the expected patch log-likelihood (EPLL), and show how this cost can be optimized iteratively using an additional step of denoising the warp error image. The results of our experiments are promising and show that patch models with higher likelihood lead to better optical flow estimation.