Structured Prompt Tuning
This work addresses the need for more flexible and effective prompt tuning methods in natural language processing, representing an incremental improvement over standard prompt tuning.
The paper tackles the problem of improving prompt tuning by proposing structured prompt tuning, which uses a hypernetwork to generate soft prompt embeddings, resulting in a gain of +1.2 to 1.5 points on the GLUE benchmark and reduced sensitivity to learning rate changes.
We propose structured prompt tuning, a simple and effective method to improve prompt tuning. Instead of prepending a sequence of tunable embeddings to the input, we generate the soft prompt embeddings through a hypernetwork. Our approach subsumes the standard prompt tuning, allows more flexibility in model design and can be applied to both single-task and multi-task training settings. Empirically, structured prompt tuning shows a gain of +1.2$~1.5 points on the GLUE benchmark and is less sensitive to the change of learning rate, compared to standard prompt tuning.