SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity
For researchers in scientific AI, this paper highlights limitations of current LLMs and evaluation methods for equation generation, but the results are incremental.
This work evaluates LLMs on generating equations from scientific text, finding moderate lexical/syntactic performance but poor semantic accuracy, with LLM-based evaluation poorly aligned with human judgments.
This work investigates the ability of large language models (LLMs) to generate mathematical equations from scientific texts. Prior work faces challenges in unstructured grounding, multi-equation dependency, and humanaligned evaluation. To this end, we construct a dataset of AI research papers, pairing contextual passages with ground-truth equations and variable descriptions. We develop an explainable equation generation workflow and evaluate it across diverse open- and closed-source LLM backbones. We introduce an evaluation protocol combining automatic metrics, LLM-based rubrics, and human judgments to assess accuracy, explainability, and human-LLM alignment. Results indicate that LLMs perform moderately on lexical- and syntactic-based similarity, while struggling with semantic accuracy. Comparisons between LLM-based evaluations and human judgments reveal limited alignment, highlighting challenges in using LLMs to assess equation quality. These findings offer insights for improving equation generation models and developing more reliable evaluation methods for scientific text. We provide code and data for reproducibility.