Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions
This addresses the problem of poor video stabilization in adverse conditions for users like videographers and autonomous systems, offering a robust solution without real data, though it is incremental as it builds on existing stabilization methods.
The paper tackles video stabilization under adverse weather conditions by proposing a synthetic-aware algorithm trained only on synthetic data, achieving the best performance across all weather conditions in stability, distortion, success rate, and cropping ratio metrics.
Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our specially generated synthetic data for training an affine transformation matrix estimator avoiding the feature extraction issues faced by current methods. Additionally, since no video stabilization datasets under adverse conditions are available, we propose the novel VSAC105Real dataset for evaluation. We compare our method to five state-of-the-art video stabilization algorithms using two benchmarks. Our results show that current approaches perform poorly in at least one weather condition, and that, even training in a small dataset with synthetic data only, we achieve the best performance in terms of stability average score, distortion score, success rate, and average cropping ratio when considering all weather conditions. Hence, our video stabilization model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.