Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
This work addresses the need for parameter-efficient fine-tuning in LLMs, offering a solution for researchers and practitioners dealing with domain-specific tasks, though it appears incremental as it builds on existing soft prompting approaches.
The paper tackled the problem of computationally expensive fine-tuning for large language models in domain-specific tasks by proposing a novel input-dependent soft prompting technique with a self-attention mechanism (ID-SPAM), which generates soft prompts based on input tokens and achieves improved zero-shot domain transfer capability compared to state-of-the-art methods.
The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.