Multilingual Large Language Models Are Not (Yet) Code-Switchers
This work addresses a gap in understanding LLM capabilities for code-switching, which is important for applications in multilingual communities, though it is incremental as it benchmarks existing models without proposing new methods.
The paper tackled the problem of evaluating multilingual large language models (LLMs) on code-switching tasks, finding that they underperform compared to fine-tuned smaller models despite showing promise in some zero-shot or few-shot scenarios.
Multilingual Large Language Models (LLMs) have recently shown great capabilities in a wide range of tasks, exhibiting state-of-the-art performance through zero-shot or few-shot prompting methods. While there have been extensive studies on their abilities in monolingual tasks, the investigation of their potential in the context of code-switching (CSW), the practice of alternating languages within an utterance, remains relatively uncharted. In this paper, we provide a comprehensive empirical analysis of various multilingual LLMs, benchmarking their performance across four tasks: sentiment analysis, machine translation, summarization and word-level language identification. Our results indicate that despite multilingual LLMs exhibiting promising outcomes in certain tasks using zero or few-shot prompting, they still underperform in comparison to fine-tuned models of much smaller scales. We argue that current "multilingualism" in LLMs does not inherently imply proficiency with code-switching texts, calling for future research to bridge this discrepancy.