LoPT: Low-Rank Prompt Tuning for Parameter Efficient Language Models
This work addresses parameter efficiency for language model adaptation, offering a more compact solution for task-specific tuning, though it is incremental as it builds on existing prompt tuning methods.
The paper tackles the problem of reducing trainable parameters in prompt tuning for language models by proposing Low-rank Prompt Tuning (LoPT), which achieves similar performance to full parameter prompt tuning while reducing parameters by a factor of 5 and outperforms state-of-the-art methods that require 10-20 times more parameters.
In prompt tuning, a prefix or suffix text is added to the prompt, and the embeddings (soft prompts) or token indices (hard prompts) of the prefix/suffix are optimized to gain more control over language models for specific tasks. This approach eliminates the need for hand-crafted prompt engineering or explicit model fine-tuning. Prompt tuning is significantly more parameter-efficient than model fine-tuning, as it involves optimizing partial inputs of language models to produce desired outputs. In this work, we aim to further reduce the amount of trainable parameters required for a language model to perform well on specific tasks. We propose Low-rank Prompt Tuning (LoPT), a low-rank model for prompts that achieves efficient prompt optimization. The proposed method demonstrates similar outcomes to full parameter prompt tuning while reducing the number of trainable parameters by a factor of 5. It also provides promising results compared to the state-of-the-art methods that would require 10 to 20 times more parameters.