LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Models
This addresses the problem of limited and inflexible datasets for evaluating and training language models on integrated reasoning skills, though it is incremental as it builds on existing synthesis and evaluation methods.
The paper tackled the challenge of joint logical-numerical reasoning in language models by introducing LogiNumSynth, a flexible synthesizer for generating controllable reasoning tasks, which revealed persistent weaknesses in LLMs and improved their performance through targeted training.
Joint logical-numerical reasoning remains a major challenge for language models, yet existing datasets rely on fixed rule sets and offer limited control over task complexity, constraining their generalizability for evaluation and training. We present LogiNumSynth, a flexible natural language problem synthesizer that synthesizes tasks requiring proficiency in joint logical reasoning (e.g., rule-based reasoning) and numerical reasoning (e.g., arithmetic computation). LogiNumSynth supports fine-grained control over reasoning world richness, logical reasoning depth, and the complexity of numerical computations, enabling flexible data synthesis across difficulty levels. We demonstrate three key contributions: (1) Synthesizer -- synthesizing fully controllable joint reasoning tasks over natural language; (2) Evaluation & Process Analysis -- evaluating both process accuracy and answer accuracy; (3) Targeted Training -- using synthesized data to enhance LLMs' reasoning performance. Experiments with multiple LLMs highlight persistent weaknesses in logical-numerical reasoning, showing that LogiNumSynth can serve as both a diagnostic tool and a source of targeted supervision for advancing integrated reasoning skills.