HyperPrompt: Prompt-based Task-Conditioning of Transformers
This addresses the challenge of efficient multi-task learning for NLP practitioners, though it is incremental as it builds on existing prompt-tuning and HyperNetwork techniques.
The authors tackled the problem of parameter-efficient multi-task learning in Transformers by introducing HyperPrompt, a method using HyperNetworks to generate task-specific prompts, achieving competitive performance with only 0.14% additional parameters.
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we propose HyperPrompt, a novel architecture for prompt-based task-conditioning of self-attention in Transformers. The hyper-prompts are end-to-end learnable via generation by a HyperNetwork. HyperPrompt allows the network to learn task-specific feature maps where the hyper-prompts serve as task global memories for the queries to attend to, at the same time enabling flexible information sharing among tasks. We show that HyperPrompt is competitive against strong multi-task learning baselines with as few as $0.14\%$ of additional task-conditioning parameters, achieving great parameter and computational efficiency. Through extensive empirical experiments, we demonstrate that HyperPrompt can achieve superior performances over strong T5 multi-task learning baselines and parameter-efficient adapter variants including Prompt-Tuning and HyperFormer++ on Natural Language Understanding benchmarks of GLUE and SuperGLUE across many model sizes.