SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformers
This work addresses the problem of efficient deployment of video diffusion transformers on edge devices, which is significant for applications requiring real-time video generation.
The authors tackled the problem of reducing memory and compute costs for video diffusion transformers, achieving results that outperform prior quantization methods and approach FP16 quality. Specifically, their proposed method, SemanticDialect, shows improved performance on video DiT models.
Diffusion Transformers (DiT) achieve strong video generation quality, but their memory and compute costs hinder edge deployment. Quantization can reduce these costs, yet existing methods often degrade video quality under high activation variation and the need to preserve semantic/temporal coherence. We propose SemanticDialect, which advances recent block-wise mixed-format quantization-selecting a per-block optimal format (a dialect) from multiple candidates (a formatbook)-by scaling the formatbook with lookup tables for quantization error and quantized values, enabling efficient per-block format selection and quantization at low online cost. We also introduce activation decomposition that reduces quantization error by re-quantizing and adding back residual errors, with attention-guided salient token selection. We further propose semantic-aware dialect assignment (SeDA) to improve quantized value consistency by sharing a sub-formatbook among semantically correlated tokens. Experiments on video DiT (VDiT) models show that SemanticDialect outperforms prior VDiT quantization methods and fine-grained block-wise format baselines, while approaching FP16 quality on Open-Sora 2.0.