Deterministic Differentiable Structured Pruning for Large Language Models
This work addresses the problem of train-test mismatch and restricted mask ranges in structured pruning for LLMs, offering a more expressive and faster converging method for researchers and practitioners aiming to reduce LLM inference costs.
This paper introduces Deterministic Differentiable Pruning (DDP), a mask-only optimization method for structured pruning of Large Language Models (LLMs) that eliminates stochasticity by directly optimizing a deterministic soft surrogate of the discrete l0 objective. DDP achieves a performance loss as small as 1% on downstream tasks for models like Qwen3-32B and Qwen3-30B-A3B, outperforming previous methods at 20% sparsity.
Structured pruning reduces LLM inference cost by removing low-importance architectural components. This can be viewed as learning a multiplicative gate for each component under an l0 sparsity constraint. Due to the discreteness of the l0 norm, prior work typically adopts stochastic hard-concrete relaxations to enable differentiable optimization; however, this stochasticity can introduce a train--test mismatch when sampled masks are discretized for deployment and restricts masks to a bounded, near-binary range. To address this, we propose Deterministic Differentiable Pruning (DDP), a mask-only optimization method that eliminates stochasticity by directly optimizing a deterministic soft surrogate of the discrete l0 objective. Compared with prior approaches, DDP offers greater expressiveness, reduced train--test mismatch, and faster convergence. We apply our method to several dense and MoE models, including Qwen3-32B and Qwen3-30B-A3B, achieving a performance loss as small as 1% on downstream tasks while outperforming previous methods at 20% sparsity. We further demonstrate end-to-end inference speedups in realistic deployment settings with vLLM.