Diagnosing and Preventing Instabilities in Recurrent Video Processing
This addresses stability issues in recurrent video processing models, which is an incremental improvement for applications requiring long-term video enhancement.
The paper tackled the problem of recurrent models failing catastrophically on long video sequences in tasks like denoising or super-resolution, and proposed a diagnostic tool and a training algorithm (SRN-C) that enforces stability without significant performance loss.
Recurrent models are a popular choice for video enhancement tasks such as video denoising or super-resolution. In this work, we focus on their stability as dynamical systems and show that they tend to fail catastrophically at inference time on long video sequences. To address this issue, we (1) introduce a diagnostic tool which produces input sequences optimized to trigger instabilities and that can be interpreted as visualizations of temporal receptive fields, and (2) propose two approaches to enforce the stability of a model during training: constraining the spectral norm or constraining the stable rank of its convolutional layers. We then introduce Stable Rank Normalization for Convolutional layers (SRN-C), a new algorithm that enforces these constraints. Our experimental results suggest that SRN-C successfully enforces stability in recurrent video processing models without a significant performance loss.