Prompt Tuning

“prompt-based PEFT methods, such as Prompt Tuning~lester2021power and Prefix-Tuning~li2021prefix, have been widely applied to Transformers but fail to adapt effectively to SSMs~galim2024parameter”

“Compared to methods like LoRA, fine-tuning approaches including Prompt Tuning, P-Tuning, and IA$^{3}$, although designed with fewer parameters, struggle to effectively capture the patterns of few samples and generalize to more samples in scenarios where data is scarce”

“it is acknowledged that training the prompt vectors in few-shot settings is prone to instability and exhibits slow convergence, making it challenging to generalize to large language models.”

“prompt tuning converges to a higher loss (left), and performs poorly compared to LoRA (right)”

“P-Tuning and Prompt Tuning are bound to the self-attention mechanism”

“However, even when learnable, adaptable prompts often struggle to capture the heterogeneity within source task distributions, possibly due to the limitations in expressive capacity imposed by their form and length”

“These methods generally utilize trainable virtual tokens within an adapter, which may not have essential semantic significance and require extensive training to acquire domain-specific knowledge efficiently. Consequently, the performance of these techniques may not be optimal, particularly when dealing with complex tasks, and extensive training is necessary to achieve optimal performance.”

“Additionally, reserving part of the sequence length for adaptation may reduce the effective input length available for the downstream task, potentially limiting performance.”

“prompt tuning underperforms even the linear probing baseline in the S3DIS 6-fold evaluation, revealing the penalty of ignoring spatial structure during fine-tuning”