Removing Cost Volumes from Optical Flow Estimators
This addresses a key bottleneck in optical flow estimation for computer vision applications, offering significant speed and memory improvements while maintaining accuracy.
The paper tackles the computational and memory inefficiency of cost volumes in optical flow estimators by introducing a training strategy that removes them, resulting in models with state-of-the-art accuracy, up to 1.2x faster inference, and up to 6x lower memory usage.
Cost volumes are used in every modern optical flow estimator, but due to their computational and space complexity, they are often a limiting factor regarding both processing speed and the resolution of input frames. Motivated by our empirical observation that cost volumes lose their importance once all other network parts of, e.g., a RAFT-based pipeline have been sufficiently trained, we introduce a training strategy that allows removing the cost volume from optical flow estimators throughout training. This leads to significantly improved inference speed and reduced memory requirements. Using our training strategy, we create three different models covering different compute budgets. Our most accurate model reaches state-of-the-art accuracy while being $1.2\times$ faster and having a $6\times$ lower memory footprint than comparable models; our fastest model is capable of processing Full HD frames at $20\,\mathrm{FPS}$ using only $500\,\mathrm{MB}$ of GPU memory.