Instruction-Following Pruning for Large Language Models
This addresses the need for more efficient and adaptable pruning techniques in large language models, offering a novel approach that is incremental but with strong specific gains.
The paper tackles the problem of static pruning in large language models by proposing a dynamic, input-dependent pruning method that adapts based on user instructions, resulting in a 3B activated model that improves over a 3B dense model by 5-8 points on math and coding tasks and rivals a 9B model's performance.
With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.