LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
This work addresses the need for standardized evaluation in LLM compression for researchers and practitioners, though it is incremental as it builds on existing quantization methods.
The paper tackles the problem of unfair comparisons in large language model quantization by introducing LLMC, a versatile compression toolkit that integrates multiple algorithms, models, and hardware, providing systematic benchmarks and insights into calibration data, algorithms, and data formats.
Recent advancements in large language models (LLMs) are propelling us toward artificial general intelligence with their remarkable emergent abilities and reasoning capabilities. However, the substantial computational and memory requirements limit the widespread adoption. Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating LLMs, albeit with potential risks to accuracy. Numerous studies have aimed to minimize the accuracy loss associated with quantization. However, their quantization configurations vary from each other and cannot be fairly compared. In this paper, we present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. LLMC integrates dozens of algorithms, models, and hardwares, offering high extensibility from integer to floating-point quantization, from LLM to vision-language (VLM) model, from fixed-bit to mixed precision, and from quantization to sparsification. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats, providing novel insights and detailed analyses for further research and practical guidance for users. Our toolkit is available at https://github.com/ModelTC/llmc.