Robust Residual Finite Scalar Quantization for Neural Compression
This addresses a fundamental limitation in neural compression for audio and image modalities, offering incremental improvements over existing methods.
The paper tackled the problem of residual magnitude decay in multi-stage Finite Scalar Quantization for neural compression by proposing Robust Residual Finite Scalar Quantization (RFSQ) with conditioning strategies, resulting in a 3.6% improvement in audio reconstruction and up to 17.4% improvement in perceptual loss on ImageNet.
Finite Scalar Quantization (FSQ) offers simplified training but suffers from residual magnitude decay in multi-stage settings, where subsequent stages receive exponentially weaker signals. We propose Robust Residual Finite Scalar Quantization (RFSQ), addressing this fundamental limitation through two novel conditioning strategies: learnable scaling factors and invertible layer normalization. Our experiments across audio and image modalities demonstrate RFSQ's effectiveness and generalizability. In audio reconstruction at 24 bits/frame, RFSQ-LayerNorm achieves 3.646 DNSMOS, a 3.6% improvement over state-of-the-art RVQ (3.518). On ImageNet, RFSQ achieves 0.102 L1 loss and 0.100 perceptual loss, with LayerNorm providing 9.7% L1 improvement and 17.4% perceptual improvement over unconditioned variants. The LayerNorm strategy consistently outperforms alternatives by maintaining normalized input statistics across stages, effectively preventing exponential magnitude decay that limits naive residual approaches. RFSQ combines FSQ's simplicity with multi-stage quantization's representational power, establishing a new standard for neural compression across diverse modalities.