ConVRT: Consistent Video Restoration Through Turbulence with Test-time Optimization of Neural Video Representations
This addresses the challenge of consistent video restoration for long-range imaging applications, offering an incremental improvement by integrating pre-trained models and test-time optimization.
The paper tackles the problem of restoring videos degraded by atmospheric turbulence, which often leads to temporal inconsistency and poor generalization, by introducing ConVRT, a self-supervised test-time optimization method that uses a neural video representation and CLIP for semantic supervision, resulting in enhanced temporal consistency and adaptability to varying real-world conditions.
tmospheric turbulence presents a significant challenge in long-range imaging. Current restoration algorithms often struggle with temporal inconsistency, as well as limited generalization ability across varying turbulence levels and scene content different than the training data. To tackle these issues, we introduce a self-supervised method, Consistent Video Restoration through Turbulence (ConVRT) a test-time optimization method featuring a neural video representation designed to enhance temporal consistency in restoration. A key innovation of ConVRT is the integration of a pretrained vision-language model (CLIP) for semantic-oriented supervision, which steers the restoration towards sharp, photorealistic images in the CLIP latent space. We further develop a principled selection strategy of text prompts, based on their statistical correlation with a perceptual metric. ConVRT's test-time optimization allows it to adapt to a wide range of real-world turbulence conditions, effectively leveraging the insights gained from pre-trained models on simulated data. ConVRT offers a comprehensive and effective solution for mitigating real-world turbulence in dynamic videos.