Scholar Feed

Parameter-efficient fine-tuning (LoRA family)

Prefix-Tuning

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Superseded baseline#10 of 1,113 most-superseded · first seen Jan 1, 2021

Superseded — cited as a baseline and beaten by newer methods

7 papers critique it · 8 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Prefix-Tuning as a baseline.

prompt-based PEFT methods, such as Prompt Tuning~lester2021power and Prefix-Tuning~li2021prefix, have been widely applied to Transformers but fail to adapt effectively to SSMs~galim2024parameter
— State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models
“but suffers the same aforementioned drawbacks.”
— PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRA
“However, they also have their limitations, such as extra inference latency in the case of adapter layers, and difficulty in optimization for prefix tuning.”
— SBoRA: Low-Rank Adaptation with Regional Weight Updates
“The use of prompts and prefix tuning techniques can pose challenges in terms of the effectiveness and interpretability of the employed prompts or prefixes. These methods generally utilize trainable virtual tokens within an adapter, which may not have essential semantic significance and require extensive training to acquire domain-specific knowledge efficiently.”
— Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning
“Optimizing prompt-based adaptations can be difficult, with performance sensitivity to the number of trainable parameters and prone to non-monotonic behavior”
— LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
“Prefix tuning hurts the model's capabilities.”
— Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
“Prefix-Tuning argue that adding few-shot demonstrations is bounded by the input length constraint of current LFMs.”
— Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors

Beaten on benchmarks

Head-to-head results where a newer method reports beating Prefix-Tuning. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.