PolyPrompt: Automating Knowledge Extraction from Multilingual Language Models with Dynamic Prompt Generation
This work addresses the problem of improving multilingual accuracy in LLMs for users in diverse linguistic settings, representing a novel method for a known bottleneck.
The paper tackled the inconsistent multilingual performance of large language models by introducing PolyPrompt, a parameter-efficient framework that enhances multilingual capabilities through dynamic prompt generation, achieving accuracy gains of 3.7% to 19.9% on the MMLU benchmark across fifteen languages.
Large language models (LLMs) showcase increasingly impressive English benchmark scores, however their performance profiles remain inconsistent across multilingual settings. To address this gap, we introduce PolyPrompt, a novel, parameter-efficient framework for enhancing the multilingual capabilities of LLMs. Our method learns a set of trigger tokens for each language through a gradient-based search, identifying the input query's language and selecting the corresponding trigger tokens which are prepended to the prompt during inference. We perform experiments on two ~1 billion parameter models, with evaluations on the global MMLU benchmark across fifteen typologically and resource diverse languages, demonstrating accuracy gains of 3.7%-19.9% compared to naive and translation-pipeline baselines.