Rethinking Layer Redundancy in Large Language Models: Calibration Objectives and Search for Depth Pruning
For practitioners pruning LLMs, this work highlights that the choice of calibration objective is critical and may outweigh search algorithm selection, challenging the assumption of universal layer redundancy.
This paper investigates depth pruning in large language models, finding that the calibration objective (e.g., perplexity vs. downstream accuracy) significantly influences which layers are deemed redundant, often more so than the choice of search algorithm. The study across three LLM families shows that different objectives yield different redundant layers and that perplexity and accuracy rankings do not consistently align.
Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work has focused on importance criteria and search algorithms, often treating layer redundancy as an inherent structural property of pretrained networks. In contrast, we adopt a \emph{functional perspective}, where redundancy is jointly influenced by the model and the evaluation objective, suggesting that a universal ranking may not be sufficient. Through an empirical study across three LLM families, two calibration objectives, and seven search algorithms, we observe that different objectives yield qualitatively different redundant layers, and that perplexity and downstream accuracy rankings do not consistently align. Under a fixed objective, however, search algorithms tend to produce similar solutions. Overall, our results suggest that the calibration objective may play a more influential role than the choice of search algorithm, indicating that further attention to objective design could be beneficial.