Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
This addresses the problem of limited multilingual reasoning capabilities in AI systems, particularly for non-English languages, by providing a data-efficient method to transfer reasoning skills from high-resource to low-resource languages, representing a strong specific gain rather than a broad paradigm shift.
The paper tackles the performance disparity in multilingual reasoning for Large Language Models by introducing PB-RLSVR, a framework that uses a high-performing English model as a pivot to generate reference responses and rewards multilingual models based on semantic equivalence, resulting in average performance improvements of 16.41% for Llama-3.1-8B-Instruct and 10.17% for Qwen3-32B on multilingual benchmarks.
While reinforcement learning has advanced the reasoning abilities of Large Language Models (LLMs), these gains are largely confined to English, creating a significant performance disparity across languages. To address this, we introduce Pivot-Based Reinforcement Learning with Semantically Verifiable Rewards (PB-RLSVR), a novel framework that enhances multilingual reasoning by circumventing the need for human-annotated data in target languages. Our approach employs a high-performing English LLM as a "pivot" model to generate reference responses for reasoning tasks. A multilingual model is then rewarded based on the semantic equivalence of its responses to the English reference, effectively transferring the pivot model's reasoning capabilities across languages. We investigate several cross-lingual semantic reward functions, including those based on embeddings and machine translation. Extensive experiments on a suite of multilingual reasoning benchmarks show that our method significantly narrows the performance gap between English and other languages, substantially outperforming traditional PPO baselines. Specifically, our PB-RLSVR framework improves the average multilingual performance of Llama-3.1-8B-Instruct and Qwen3-32B by 16.41% and 10.17%, respectively, demonstrating a powerful and data-efficient approach to building truly multilingual reasoning agents.