Effectively Prompting Small-sized Language Models for Cross-lingual Tasks via Winning Tickets
This work addresses the problem of prompting small models for cross-lingual tasks, particularly in low-resource settings, offering a simpler and more efficient method compared to prior approaches that rely on external tools.
The paper tackles the limited performance of soft prompt methods on small language models by introducing the Lottery Ticket Prompt-learning (LTP) framework, which integrates winning tickets with soft prompts to achieve better results on cross-lingual tasks, outperforming baselines by updating only 20% of parameters.
Current soft prompt methods yield limited performance when applied to small-sized models (fewer than a billion parameters). Deep prompt-tuning, which entails prepending parameters in each layer for enhanced efficacy, presents a solution for prompting small-sized models, albeit requiring carefully designed implementation. In this paper, we introduce the Lottery Ticket Prompt-learning (LTP) framework that integrates winning tickets with soft prompts. The LTP offers a simpler implementation and requires only a one-time execution. We demonstrate LTP on cross-lingual tasks, where prior works rely on external tools like human-designed multilingual templates and bilingual dictionaries, which may not be feasible in a low-resource regime. Specifically, we select a subset of parameters that have been changed the most during the fine-tuning with the Masked Language Modeling objective. Then, we prepend soft prompts to the original pre-trained language model and only update the selected parameters together with prompt-related parameters when adapting to the downstream tasks. We verify the effectiveness of our LTP framework on cross-lingual tasks, specifically targeting low-resource languages. Our approach outperforms the baselines by only updating 20\% of the original parameters.