Eigenpruning: an Interpretability-Inspired PEFT Method
This provides a computationally efficient method for enhancing task-specific performance in LLMs, though it appears incremental as it builds on existing interpretability-inspired pruning techniques.
The paper tackles the problem of improving LLM performance on specific tasks by removing singular values from weight matrices, achieving a test accuracy increase from 13.75% to 97.50% in integer multiplication with the Phi-2 model.
We introduce eigenpruning, a method that removes singular values from weight matrices in an LLM to improve its performance in a particular task. This method is inspired by interpretability methods designed to automatically find subnetworks of a model which solve a specific task. In our tests, the pruned model outperforms the original model by a large margin, while only requiring minimal computation to prune the weight matrices. In the case of a small synthetic task in integer multiplication, the Phi-2 model can improve its accuracy in the test set from 13.75% to 97.50%. Interestingly, these results seem to indicate the existence of a computation path that can solve the task very effectively, but it was not being used by the original model. Finally, we publicly release our implementation.