On the Limits of Layer Pruning for Generative Reasoning in LLMs
This work addresses the challenge of effectively compressing LLMs for generative reasoning under constrained post-training regimes, though it is incremental in characterizing practical limits.
The paper tackled the problem of layer pruning in large language models (LLMs) causing severe degradation in generative reasoning tasks, finding that a mitigation strategy with supervised finetuning recovered up to 90% of baseline performance on classification and improved generative benchmarks by 20-30 percentage points, but recovery remained limited at higher pruning ratios.
Recent works have shown that layer pruning can compress large language models (LLMs) while retaining strong performance on classification benchmarks with little or no finetuning. However, existing pruning techniques often suffer severe degradation on generative reasoning tasks. Through a systematic study across multiple model families, we find that tasks requiring multi-step reasoning are particularly sensitive to depth reduction. Beyond surface-level text degeneration, we observe degradation of critical algorithmic capabilities, including arithmetic computation for mathematical reasoning and balanced parenthesis generation for code synthesis. Under realistic post-training constraints, without access to pretraining-scale data or compute, we evaluate a simple mitigation strategy based on supervised finetuning with Self-Generated Responses. This approach achieves strong recovery on classification tasks, retaining up to 90\% of baseline performance, and yields substantial gains of up to 20--30 percentage points on generative benchmarks compared to prior post-pruning techniques. Crucially, despite these gains, recovery for generative reasoning remains fundamentally limited relative to classification tasks and is viable primarily at lower pruning ratios. Overall, we characterize the practical limits of layer pruning for generative reasoning and provide guidance on when depth reduction can be applied effectively under constrained post-training regimes.