Minimal Pair-Based Evaluation of Code-Switching
This provides a scalable evaluation methodology for code-switching in LLMs, addressing a gap in language coverage and CS phenomena, but it is incremental as it builds on existing minimal pair concepts.
The paper tackles the problem of evaluating how well large language models (LLMs) mimic bilingual code-switching (CS) by proposing a minimal pair-based intervention, showing that larger models more consistently assign higher probability to naturally occurring CS sentences, with the largest differences for closed-class words.
There is a lack of an evaluation methodology that estimates the extent to which large language models (LLMs) use code-switching (CS) in the same way as bilinguals. Existing methods do not have wide language coverage, fail to account for the diverse range of CS phenomena, or do not scale. We propose an intervention based on minimal pairs of CS. Each minimal pair contains one naturally occurring CS sentence and one minimally manipulated variant. We collect up to 1,000 such pairs each for 11 language pairs. Our human experiments show that, for every language pair, bilinguals consistently prefer the naturally occurring CS sentence. Meanwhile our experiments with current LLMs show that the larger the model, the more consistently it assigns higher probability to the naturally occurring CS sentence than to the variant. In accordance with theoretical claims, the largest probability differences arise in those pairs where the manipulated material consisted of closed-class words.