Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models
This study addresses a gap in evaluating multilingual LLMs for long-context information retrieval, which is crucial for improving model robustness in diverse language settings, though it is incremental as it extends existing evaluation methods.
The paper tackled the problem of evaluating long-context capabilities of multilingual large language models (LLMs) by introducing the MLNeedle test, revealing that performance varies significantly with language and needle position, with lowest results for non-English languages and middle positions, and no models achieved satisfactory cross-lingual retrieval as context length increased.
While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model's ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of $8k$ tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.