Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning
This work addresses the challenge of enhancing multilingual reasoning in LLMs by leveraging complementary strengths across languages, though it is incremental as it builds on existing reward modeling techniques.
The study tackled the problem of varying reasoning abilities across languages in multilingual LLMs by training a cross-lingual reward model to rank responses, resulting in substantial improvements in mathematical reasoning performance, including benefits for high-resource languages like English under low sampling budgets.
While the reasoning abilities of large language models (LLMs) continue to advance, it remains unclear how such ability varies across languages in multilingual LLMs and whether different languages produce reasoning paths that complement each other. To investigate this question, we train a reward model to rank generated responses for a given question across languages. Our results show that our cross-lingual reward model substantially improves mathematical reasoning performance compared to using reward modeling within a single language, benefiting even high-resource languages. While English often exhibits the highest performance in multilingual models, we find that cross-lingual sampling particularly benefits English under low sampling budgets. Our findings reveal new opportunities to improve multilingual reasoning by leveraging the complementary strengths of diverse languages.