Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition
This work addresses the problem of efficient video processing for edge computing applications, offering a novel compression method that is incremental in improving existing semantic communication approaches.
The paper tackles computational offloading for real-time video action recognition by proposing a spatiotemporal attention-based autoencoder (STAE) architecture that compresses video data, achieving 104x compression on the HMDB51 dataset with only 5% accuracy loss and outperforming the baseline DeepISC in speed and accuracy under time-varying wireless channels.
This paper studies the computational offloading of video action recognition in edge computing. To achieve effective semantic information extraction and compression, following semantic communication we propose a novel spatiotemporal attention-based autoencoder (STAE) architecture, including a frame attention module and a spatial attention module, to evaluate the importance of frames and pixels in each frame. Additionally, we use entropy encoding to remove statistical redundancy in the compressed data to further reduce communication overhead. At the receiver, we develop a lightweight decoder that leverages a 3D-2D CNN combined architecture to reconstruct missing information by simultaneously learning temporal and spatial information from the received data to improve accuracy. To fasten convergence, we use a step-by-step approach to train the resulting STAE-based vision transformer (ViT_STAE) models. Experimental results show that ViT_STAE can compress the video dataset HMDB51 by 104x with only 5% accuracy loss, outperforming the state-of-the-art baseline DeepISC. The proposed ViT_STAE achieves faster inference and higher accuracy than the DeepISC-based ViT model under time-varying wireless channel, which highlights the effectiveness of STAE in guaranteeing higher accuracy under time constraints.