Soft Language Prompts for Language Transfer
This work addresses cross-lingual NLP applications, particularly for low-resource languages, but is incremental as it builds on existing parameter-efficient fine-tuning methods.
This study tackled the challenge of cross-lingual knowledge transfer in NLP by exploring combinations of adapters and soft prompts, finding that soft language prompts with task adapters often outperform other configurations across 16 languages, including 10 mid- and low-resource ones.
Cross-lingual knowledge transfer, especially between high- and low-resource languages, remains challenging in natural language processing (NLP). This study offers insights for improving cross-lingual NLP applications through the combination of parameter-efficient fine-tuning methods. We systematically explore strategies for enhancing cross-lingual transfer through the incorporation of language-specific and task-specific adapters and soft prompts. We present a detailed investigation of various combinations of these methods, exploring their efficiency across 16 languages, focusing on 10 mid- and low-resource languages. We further present to our knowledge the first use of soft prompts for language transfer, a technique we call soft language prompts. Our findings demonstrate that in contrast to claims of previous work, a combination of language and task adapters does not always work best; instead, combining a soft language prompt with a task adapter outperforms most configurations in many cases.