AutoCompress: Critical Layer Isolation for Efficient Transformer Compression
For practitioners deploying transformer models under resource constraints, this work provides a compression method that preserves task-critical information in early layers, yielding significantly better performance than uniform compression at similar sizes.
AutoCompress introduces Critical Layer Isolation (CLI), which protects Layer 0 at full dimensionality while compressing intermediate layers, achieving 2.47x compression on GPT-2 Medium (143.8M parameters) with 204.5 perplexity on WikiText-103, compared to 571.8 perplexity for a uniform bottleneck baseline.
We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.