Prompting PaLM for Translation: Assessing Strategies and Performance
This work provides insights into translation strategies for large language models, but it is incremental as it builds on existing assessments with updated metrics and datasets.
The study assessed the machine translation capabilities of the PaLM language model using few-shot prompting, finding that example quality is the most important factor, and while PaLM's performance is impressive, it still lags behind state-of-the-art supervised systems.
Large language models (LLMs) that have been trained on multilingual but not parallel text exhibit a remarkable ability to translate between languages. We probe this ability in an in-depth study of the pathways language model (PaLM), which has demonstrated the strongest machine translation (MT) performance among similarly-trained LLMs to date. We investigate various strategies for choosing translation examples for few-shot prompting, concluding that example quality is the most important factor. Using optimized prompts, we revisit previous assessments of PaLM's MT capabilities with more recent test sets, modern MT metrics, and human evaluation, and find that its performance, while impressive, still lags that of state-of-the-art supervised systems. We conclude by providing an analysis of PaLM's MT output which reveals some interesting properties and prospects for future work.