Scholar Feed

Parameter-efficient fine-tuning (LoRA family)

BitFit

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Heavily superseded#5 of 1,113 most-superseded · first seen Jun 18, 2021

Heavily superseded — a standard baseline that newer methods routinely beat

7 papers critique it · 28 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites BitFit as a baseline.

Yet, because weight matrices are not changed %the bias terms remain constant across inputs % %(see Section~sec: method), %variation between samples are not taken into account during fine-tuning. As a result, % they may not reach the performance achieved by other methods, such as Low-Rank Adaptation (LoRA) hu2021lora, pu2023empirical.
— 1LoRA: Summation Compression for Very Low-Rank Adaptation
“Unlike the previous literature like Liu2022FewShotPF or BenZaken2021BitFitSP, we introduce a novel prompt-aware mechanism to the PEFT method.”
— PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment
“This includes methods previously considered less effective, such as BitFit~zaken2022bitfit, which FT only the bias terms of the frozen backbone.”
— Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition
“Research based on sparse methods zaken2021bitfit optimizes the bias parameter to reduce cost and improve the performance of the model during specific tasks but faces problems in dealing with real data.”
— Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models
“BitFit is susceptible to catastrophic forgetting when the target task has a large zero-shot silhouette score”
— Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIP
“We hypothesize, however, that the bias may not necessarily be the optimal component of BERT for parameter-efficient fine-tuning, and similar/better performance could be obtained by training a smaller number of parameters if the optimal component is chosen.”
— LayerNorm: A key component in parameter-efficient fine-tuning
“BitFit method zaken2021bitfit updates only the bias parameters, resulting in a substantial reduction in the number of trainable parameters, but at the cost of suboptimal performance.”
— PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Beaten on benchmarks

Head-to-head results where a newer method reports beating BitFit. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.