Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
For practitioners fine-tuning LLMs, this work offers a simple, low-cost sparse PEFT method that leverages existing pruning signals, though improvements are incremental over existing approaches.
The paper proposes Super, a sparse parameter-efficient fine-tuning method that reuses pruning saliency signals to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. On Math17K arithmetic tasks with Llama-3.2-1B and Meta-Llama-3-8B, the best variants achieve the highest average accuracy among tested configurations.
Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.