Structural Sensitivity in Compressed Transformers: Error Propagation, Lyapunov Stability, and Formally Verified Bounds
This work addresses the problem of unpredictable performance degradation in compressed transformers for AI practitioners, providing insights into sensitivity patterns and formal verification methods, though it is incremental in refining compression analysis.
The study identified that transformer compression sensitivity varies drastically, with a single matrix in GPT-2 Small increasing perplexity by 20,000x when compressed, and found a consistent hierarchy where early-layer MLP up-projections are highly sensitive while value projections are robust. It used Lyapunov stability theory to show that residual connections contract errors, and formalized error bounds with ten machine-checked theorems, validated across over 14,000 configurations.
A single matrix out of 468 in GPT-2 Small can increase perplexity by 20,000x when compressed, revealing that transformer compression sensitivity spans five orders of magnitude. We map this sensitivity landscape across five architectures (117M-8B parameters), finding a consistent hierarchy: early-layer MLP up-projections are catastrophically sensitive while value projections compress nearly for free. This hierarchy is stable across compression levels, evaluation scales (2K-51K tokens), and datasets (WikiText-103, C4). Using Lyapunov stability theory, we show that residual connections contract compression errors by growing the hidden state faster than the error. Error contraction is necessary but not sufficient for compression tolerance: architecture-specific redundancy plays an equally important role, as demonstrated by the hybrid LFM2-2.6B degrading only 7x despite higher amplification than the fully-contracting GPT-2 Small (120x). Ten machine-checked Lean 4 theorems formalize per-matrix error bounds with no sorry markers; all bounds produce zero violations across 14,040+ configurations. We validate with downstream task evaluation (HellaSwag, ARC-Easy, Winogrande), activation-aware pruning on two architectures, and a Compression Fragility Index that rank-orders model robustness.