Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic
This addresses the challenge of improving perceptual quality in compressed video enhancement for applications like streaming or video processing, though it is incremental as it builds on existing adversarial training methods.
The paper tackles the problem of selecting perceptually relevant layers from deep feature quality metrics for video enhancement, resulting in up to 10% FID and 15% KID performance improvements.
Image based Deep Feature Quality Metrics (DFQMs) have been shown to better correlate with subjective perceptual scores over traditional metrics. The fundamental focus of these DFQMs is to exploit internal representations from a large scale classification network as the metric feature space. Previously, no attention has been given to the problem of identifying which layers are most perceptually relevant. In this paper we present a new method for selecting perceptually relevant layers from such a network, based on a neuroscience interpretation of layer behaviour. The selected layers are treated as a hyperparameter to the critic network in a W-GAN. The critic uses the output from these layers in the preliminary stages to extract perceptual information. A video enhancement network is trained adversarially with this critic. Our results show that the introduction of these selected features into the critic yields up to 10% (FID) and 15% (KID) performance increase against other critic networks that do not exploit the idea of optimised feature selection.