Cross Modal Compression: Towards Human-comprehensible Semantic Compression
This addresses the need for efficient semantic monitoring and machine analysis in compression, though it appears incremental as it builds on existing cross-modal translation methods.
The paper tackles the problem of compressing visual data by preserving semantic fidelity rather than signal fidelity, proposing a cross-modal compression framework that transforms images/videos into compact human-comprehensible domains like text or sketches, achieving an ultrahigh compression ratio and better performance than JPEG.
Traditional image/video compression aims to reduce the transmission/storage cost with signal fidelity as high as possible. However, with the increasing demand for machine analysis and semantic monitoring in recent years, semantic fidelity rather than signal fidelity is becoming another emerging concern in image/video compression. With the recent advances in cross modal translation and generation, in this paper, we propose the cross modal compression~(CMC), a semantic compression framework for visual data, to transform the high redundant visual data~(such as image, video, etc.) into a compact, human-comprehensible domain~(such as text, sketch, semantic map, attributions, etc.), while preserving the semantic. Specifically, we first formulate the CMC problem as a rate-distortion optimization problem. Secondly, we investigate the relationship with the traditional image/video compression and the recent feature compression frameworks, showing the difference between our CMC and these prior frameworks. Then we propose a novel paradigm for CMC to demonstrate its effectiveness. The qualitative and quantitative results show that our proposed CMC can achieve encouraging reconstructed results with an ultrahigh compression ratio, showing better compression performance than the widely used JPEG baseline.