Solving Probability and Statistics Problems by Program Synthesis
This work addresses the automation of solving complex probability problems for educational or research purposes, but it is incremental as it applies existing LLM capabilities to a new dataset.
The authors tackled university-level probability and statistics problems by using OpenAI's Codex for program synthesis, transforming course questions into programming tasks and executing the generated code to compute solutions, with a focus on prompt engineering to achieve correct results.
We solve university level probability and statistics questions by program synthesis using OpenAI's Codex, a Transformer trained on text and fine-tuned on code. We transform course problems from MIT's 18.05 Introduction to Probability and Statistics and Harvard's STAT110 Probability into programming tasks. We then execute the generated code to get a solution. Since these course questions are grounded in probability, we often aim to have Codex generate probabilistic programs that simulate a large number of probabilistic dependencies to compute its solution. Our approach requires prompt engineering to transform the question from its original form to an explicit, tractable form that results in a correct program and solution. To estimate the amount of work needed to translate an original question into its tractable form, we measure the similarity between original and transformed questions. Our work is the first to introduce a new dataset of university-level probability and statistics problems and solve these problems in a scalable fashion using the program synthesis capabilities of large language models.