An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)
This provides an independent evaluation of ChatGPT's capabilities on math problems, highlighting performance variability that could impact users relying on it for educational or problem-solving tasks, though it is incremental as it applies existing methods to new data.
The study evaluated ChatGPT's performance on mathematical word problems from the DRAW-1K dataset, finding that its failure rate increased from 20% when showing work to 84% when not, with failure probability rising linearly with the number of addition and subtraction operations.
We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.