Multimodal generative semantic communication based on latent diffusion model
This work addresses the need for robust environmental data gathering and decision-making in emergencies, offering an incremental improvement over prior semantic communication methods by incorporating multimodal inputs.
The paper tackles the problem of limited decision accuracy in emergencies due to traditional single-modality semantic communication by introducing mm-GESCO, a multimodal generative framework that fuses visible and infrared image data to generate semantic segmentation maps, achieving a compression ratio of up to 200 times and outperforming existing frameworks in tasks like object classification and detection.
In emergencies, the ability to quickly and accurately gather environmental data and command information, and to make timely decisions, is particularly critical. Traditional semantic communication frameworks, primarily based on a single modality, are susceptible to complex environments and lighting conditions, thereby limiting decision accuracy. To this end, this paper introduces a multimodal generative semantic communication framework named mm-GESCO. The framework ingests streams of visible and infrared modal image data, generates fused semantic segmentation maps, and transmits them using a combination of one-hot encoding and zlib compression techniques to enhance data transmission efficiency. At the receiving end, the framework can reconstruct the original multimodal images based on the semantic maps. Additionally, a latent diffusion model based on contrastive learning is designed to align different modal data within the latent space, allowing mm-GESCO to reconstruct latent features of any modality presented at the input. Experimental results demonstrate that mm-GESCO achieves a compression ratio of up to 200 times, surpassing the performance of existing semantic communication frameworks and exhibiting excellent performance in downstream tasks such as object classification and detection.