AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression
This provides a more accessible and efficient solution for researchers and practitioners needing to compress and deploy large models, though it is incremental as it consolidates existing algorithms into a unified toolkit.
The authors tackled the problem of compressing large models for industrial deployment by introducing AngelSlim, a toolkit that integrates quantization, speculative decoding, and pruning, achieving results such as a 2-bit large model (HY-1.8B-int2), 1.8x to 2.0x throughput gains, and reduced Time-to-First-Token in long-context scenarios.
This technical report introduces AngelSlim, a comprehensive and versatile toolkit for large model compression developed by the Tencent Hunyuan team. By consolidating cutting-edge algorithms, including quantization, speculative decoding, token pruning, and distillation. AngelSlim provides a unified pipeline that streamlines the transition from model compression to industrial-scale deployment. To facilitate efficient acceleration, we integrate state-of-the-art FP8 and INT8 Post-Training Quantization (PTQ) algorithms alongside pioneering research in ultra-low-bit regimes, featuring HY-1.8B-int2 as the first industrially viable 2-bit large model. Beyond quantization, we propose a training-aligned speculative decoding framework compatible with multimodal architectures and modern inference engines, achieving 1.8x to 2.0x throughput gains without compromising output correctness. Furthermore, we develop a training-free sparse attention framework that reduces Time-to-First-Token (TTFT) in long-context scenarios by decoupling sparse kernels from model architectures through a hybrid of static patterns and dynamic token selection. For multimodal models, AngelSlim incorporates specialized pruning strategies, namely IDPruner for optimizing vision tokens via Maximal Marginal Relevance and Samp for adaptive audio token merging and pruning. By integrating these compression strategies from low-level implementations, AngelSlim enables algorithm-focused research and tool-assisted deployment.