How can representation dimension dominate structurally pruned LLMs?
This addresses the challenge of efficient pruning for large language models, offering a method to predict performance without costly evaluations, though it is incremental as it builds on existing pruning techniques like SliceGPT.
The paper tackles the problem of understanding how model performance varies with different subnetwork extractions in structurally pruned LLMs, finding that representation dimension dominates linear transformations and predictions, and provides analytical relations to predict pruned model performance without evaluation, validated on Llama-3-8B-Instruct and Phi-3-mini-4k-Instruct.
Pruning assumes a subnetwork exists in the original deep neural network, which can achieve comparative model performance with less computation than the original. However, it is unclear how the model performance varies with the different subnetwork extractions. In this paper, we choose the representation dimension (or embedding dimension, model dimension, the dimension of the residual stream in the relevant literature) as the entry point to this issue. We investigate the linear transformations in the LLM transformer blocks and consider a specific structured pruning approach, SliceGPT, to extract the subnetworks of different representation dimensions. We mechanistically analyse the activation flow during the model forward passes, and find the representation dimension dominates the linear transformations, model predictions, and, finally, the model performance. Explicit analytical relations are given to calculate the pruned model performance (perplexity and accuracy) without actual evaluation, and are empirically validated with Llama-3-8B-Instruct and Phi-3-mini-4k-Instruct.