Late Breaking Results: Quamba-SE: Soft-edge Quantizer for Activations in State Space Models
This work addresses efficient deployment of SSMs for practitioners by improving quantization performance, though it is incremental over prior methods like Quamba.
The paper tackles activation quantization in State Space Models by introducing Quamba-SE, a soft-edge quantizer that uses adaptive scales to preserve outlier information, resulting in up to +2.68% improvement on individual benchmarks and +0.83% average accuracy gain across 6 datasets.
We propose Quamba-SE, a soft-edge quantizer for State Space Model (SSM) activation quantization. Unlike existing methods, using standard INT8 operation, Quamba-SE employs three adaptive scales: high-precision for small values, standard scale for normal values, and low-precision for outliers. This preserves outlier information instead of hard clipping, while maintaining precision for other values. We evaluate on Mamba- 130M across 6 zero-shot benchmarks. Results show that Quamba- SE consistently outperforms Quamba, achieving up to +2.68% on individual benchmarks and up to +0.83% improvement in the average accuracy of 6 datasets.