Locality-Aware Redundancy Pruning for LLM Depth Compression
For LLM practitioners, LoRP offers a more effective one-shot depth pruning technique that adapts to varying inter-layer redundancy patterns across architectures.
LoRP introduces a training-free depth pruning method for LLMs that uses representation locality to identify and remove redundant layers, achieving improved perplexity and downstream task accuracy across diverse model families.
Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy.