UniMIC: Towards Universal Multi-modality Perceptual Image Compression
This work addresses the challenge of efficient multi-modality image compression for applications requiring diverse codecs, though it appears incremental by building on existing methods like stable diffusion.
The paper tackles the problem of unifying rate-distortion-perception optimization for multiple image codecs by introducing UniMIC, a framework that uses a visual codec repository, multi-grained textual coding, and a universal perception compensator, achieving significant improvements in RDP optimization across different codecs and bitrates.
We present UniMIC, a universal multi-modality image compression framework, intending to unify the rate-distortion-perception (RDP) optimization for multiple image codecs simultaneously through excavating cross-modality generative priors. Unlike most existing works that need to design and optimize image codecs from scratch, our UniMIC introduces the visual codec repository, which incorporates amounts of representative image codecs and directly uses them as the basic codecs for various practical applications. Moreover, we propose multi-grained textual coding, where variable-length content prompt and compression prompt are designed and encoded to assist the perceptual reconstruction through the multi-modality conditional generation. In particular, a universal perception compensator is proposed to improve the perception quality of decoded images from all basic codecs at the decoder side by reusing text-assisted diffusion priors from stable diffusion. With the cooperation of the above three strategies, our UniMIC achieves a significant improvement of RDP optimization for different compression codecs, e.g., traditional and learnable codecs, and different compression costs, e.g., ultra-low bitrates. The code will be available in https://github.com/Amygyx/UniMIC .