Compressed code: the hidden effects of quantization and distillation on programming tokens
This work addresses the problem of maintaining code generation quality in compressed LLMs for developers and researchers, though it is incremental in analyzing existing methods on new data.
The paper investigated how programming language tokens are encoded in LLM tokenizers and how optimization techniques like quantization and distillation affect token-level representations and code generation quality, revealing critical insights and empirically-validated guidelines.
Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments.