Is Prefix-Tuning superseded?

Prefix-Tuning (Parameter-efficient fine-tuning (LoRA family)): superseded — cited as a baseline and beaten by newer methods. 7 paper(s) critique it, 8 beat it on benchmarks — #10 of 1113 most-superseded. Sub-problem: cluster led by LoRA. Newer alternatives in the same sub-problem include Balanced LoRA, FedSmoothLoRA, FuRA, LoRA-Over, Hybrid-LoRA.

Method Drift›Parameter-efficient fine-tuning (LoRA family)

Superseded baseline#10 of 1,113 most-superseded

Prefix-Tuning

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Parameter-efficient fine-tuning (LoRA family) · first seen Jan 1, 2021

superseded — cited as a baseline and beaten by newer methods

7 papers critique it · 8 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Prefix-Tuning as a baseline.

“prompt-based PEFT methods, such as Prompt Tuning~lester2021power and Prefix-Tuning~li2021prefix, have been widely applied to Transformers but fail to adapt effectively to SSMs~galim2024parameter”
— State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
“but suffers the same aforementioned drawbacks.”
— PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
“However, they also have their limitations, such as extra inference latency in the case of adapter layers, and difficulty in optimization for prefix tuning.”
— SBoRA: Low-Rank Adaptation with Regional Weight Updates
“The use of prompts and prefix tuning techniques can pose challenges in terms of the effectiveness and interpretability of the employed prompts or prefixes. These methods generally utilize trainable virtual tokens within an adapter, which may not have essential semantic significance and require extensive training to acquire domain-specific knowledge efficiently.”
— Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning
“Optimizing prompt-based adaptations can be difficult, with performance sensitivity to the number of trainable parameters and prone to non-monotonic behavior”
— LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
“Prefix tuning hurts the model's capabilities.”
— Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
“Prefix-Tuning argue that adding few-shot demonstrations is bounded by the input length constraint of current LFMs.”
— Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors

Beaten on benchmarks

Head-to-head results where a newer method reports beating Prefix-Tuning. Values are copied from the source paper's tables — verify against the cited paper.

State-offset Tuning beats Prefix-Tuning · Spider (All) [Mamba 1.4B]
57.4 vs 39.65183794
State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
State-offset Tuning beats Prefix-Tuning · GLUE (Avg.) [Mamba 1.4B]
78.5 vs 68.6
State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
NS LoRA (3) beats Prefix-Tuning · Accuracy [GSM8K, Llama-3.1-8B-Instruct]
80.43 vs 77.79
Neurosymbolic LoRA: Why and When to Tune Weights vs. Rewrite Prompts
Adapter beats Prefix-Tuning · Average [all tasks]
64.4 vs 51.1
PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization
LoRA beats Prefix-Tuning · Acc w/ Reduced Params [111K adjusted params]
66.2 vs 61.5
PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization
LoRA-LinProj beats Prefix-Tuning · Avg. Score [Mamba model, GLUE]
81.2 vs 68.6
Parameter-Efficient Fine-Tuning of State Space Models
LoRA-LinProj beats Prefix-Tuning · METEOR [Mamba model, DART]
70.9 vs 66.6
Parameter-Efficient Fine-Tuning of State Space Models
LoRA-LinProj beats Prefix-Tuning · BLEU [Mamba model, DART]
49.5 vs 42.5
Parameter-Efficient Fine-Tuning of State Space Models
LoRA-LinProj beats Prefix-Tuning · Acc. [Mamba model, Spider]
57.5 vs 39.7
Parameter-Efficient Fine-Tuning of State Space Models
LoRA-LinProj beats Prefix-Tuning · Acc. [Mamba model, CIFAR-10]
61.0 vs 41.0
Parameter-Efficient Fine-Tuning of State Space Models
LoRA-Both beats Prefix-Tuning · Acc. [Mamba model, CelebA]
89.8 vs 86.5
Parameter-Efficient Fine-Tuning of State Space Models
SK-Tuning (Prompt) beats Prefix-Tuning · Avg [RoBERTa Base (RoB_B)]
85.45 vs 80.88
Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning

Newer alternatives

Recent methods in the same sub-problem, not yet superseded in the knowledge base.