Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoning
This work addresses the challenge of improving LLM performance in formal reasoning tasks and low-resource language applications, but it is incremental as it focuses on benchmarking and fine-tuning optimizations.
The researchers tackled the problem of evaluating large language models' logical reasoning in low-resource settings by introducing Rosetta-PL, a benchmark based on translating logical propositions, and found that preserving logical relationships in translation significantly boosts precision, with accuracy plateauing beyond about 20,000 training samples.
Large Language Models (LLMs) are primarily trained on high-resource natural languages, limiting their effectiveness in low-resource settings and in tasks requiring deep logical reasoning. This research introduces Rosetta-PL, a benchmark designed to evaluate LLMs' logical reasoning and generalization capabilities in a controlled environment. We construct Rosetta-PL by translating a dataset of logical propositions from Lean into a custom logical language, which is then used to fine-tune an LLM (e.g., GPT-4o). Our experiments analyze the impact of the size of the dataset and the translation methodology on the performance of the model. Our results indicate that preserving logical relationships in the translation process significantly boosts precision, with accuracy plateauing beyond roughly 20,000 training samples. These insights provide valuable guidelines for optimizing LLM training in formal reasoning tasks and improving performance in various low-resource language applications.