Convolutional Block Design for Learned Fractional Downsampling
This addresses the need for fractional resizing in image and video processing applications, such as adaptive bitrate streaming, but is incremental as it adapts existing CNN architectures rather than introducing a fundamentally new approach.
The paper tackles the problem of limited integer scaling factors in convolutional neural networks by proposing a new building block that enables fractional downsampling, achieving improved coding efficiency over the Lanczos algorithm in video streaming with gains in PSNR, SSIM, and VMAF metrics.
The layers of convolutional neural networks (CNNs) can be used to alter the resolution of their inputs, but the scaling factors are limited to integer values. However, in many image and video processing applications, the ability to resize by a fractional factor would be advantageous. One example is conversion between resolutions standardized for video compression, such as from 1080p to 720p. To solve this problem, we propose an alternative building block, formulated as a conventional convolutional layer followed by a differentiable resizer. More concretely, the convolutional layer preserves the resolution of the input, while the resizing operation is fully handled by the resizer. In this way, any CNN architecture can be adapted for non-integer resizing. As an application, we replace the resizing convolutional layer of a modern deep downsampling model by the proposed building block, and apply it to an adaptive bitrate video streaming scenario. Our experimental results show that an improvement in coding efficiency over the conventional Lanczos algorithm is attained, in terms of PSNR, SSIM, and VMAF on test videos.