Parameter-efficient fine-tuning (LoRA family)

Most-superseded baselines

Ranked by how many distinct papers critique or beat each method. These are the standard baselines newer work routinely measures against.

1. 1

   LoRA· LoRA

   LoRA: Low-Rank Adaptation of Large Language Models

   194 papers critique it · 201 beat it on benchmarks

2. 2

   AdaLoRA· LoRA

   AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

   29 papers critique it · 54 beat it on benchmarks

3. 3

   DoRA· LoRA

   DoRA: Weight-Decomposed Low-Rank Adaptation

   16 papers critique it · 66 beat it on benchmarks

4. 4

   PiSSA· LoRA

   PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

   14 papers critique it · 30 beat it on benchmarks

5. 5

   BitFit· LoRA

   BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

   7 papers critique it · 28 beat it on benchmarks

6. 6

   Adapter· LoRA

   Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

   10 papers critique it · 23 beat it on benchmarks

7. 7

   FFA-LoRA· FFA-LoRA

   9 papers critique it · 17 beat it on benchmarks

8. 8

   MoELoRA· LoRA

   MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

   11 papers critique it · 10 beat it on benchmarks

9. 9

   QLoRA· LoRA

   QLoRA: Efficient Finetuning of Quantized LLMs

   5 papers critique it · 14 beat it on benchmarks

10. 10

    Prefix-Tuning· LoRA

    Prefix-Tuning: Optimizing Continuous Prompts for Generation

    7 papers critique it · 8 beat it on benchmarks

11. 11

    Prompt Tuning· LoRA

    9 papers critique it · 6 beat it on benchmarks

12. 12

    LoRA+· LoRA

    LoRA: Low-Rank Adaptation of Large Language Models

    2 papers critique it · 12 beat it on benchmarks