Towards Modality Transferable Visual Information Representation with Optimal Model Compression
This work addresses the need for improved visual signal representation in image/video applications, offering a novel approach that is incremental in nature.
The paper tackles the problem of compactly representing visual signals by introducing a modality transferable scheme that transforms visual signals into another modality for enhanced representation, achieving significantly better representation capability as demonstrated on the versatile video coding standard.
Compactly representing the visual signals is of fundamental importance in various image/video-centered applications. Although numerous approaches were developed for improving the image and video coding performance by removing the redundancies within visual signals, much less work has been dedicated to the transformation of the visual signals to another well-established modality for better representation capability. In this paper, we propose a new scheme for visual signal representation that leverages the philosophy of transferable modality. In particular, the deep learning model, which characterizes and absorbs the statistics of the input scene with online training, could be efficiently represented in the sense of rate-utility optimization to serve as the enhancement layer in the bitstream. As such, the overall performance can be further guaranteed by optimizing the new modality incorporated. The proposed framework is implemented on the state-of-the-art video coding standard (i.e., versatile video coding), and significantly better representation capability has been observed based on extensive evaluations.