MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training
This addresses a domain-specific problem for researchers and practitioners in AI and scientific fields by improving language models' handling of mathematical content, though it is incremental as it builds on existing dataset generation methods.
The paper tackles the challenge of language models struggling with mathematical notation by developing MAMUT, a framework that generates equivalent and falsified versions of mathematical formulas in LaTeX to create specialized training datasets, resulting in new state-of-the-art performance on mathematical retrieval tasks.
Mathematical formulas are a fundamental and widely used component in various scientific fields, serving as a universal language for expressing complex concepts and relationships. While state-of-the-art transformer models excel in processing and understanding natural language, they encounter challenges with mathematical notation, which involves a complex structure and diverse representations. This study focuses on the development of specialized training datasets to enhance the encoding of mathematical content. We introduce Math Mutator (MAMUT), a framework capable of generating equivalent and falsified versions of a given mathematical formula in LaTeX notation, effectively capturing the mathematical variety in notation of the same concept. Based on MAMUT, we have generated four large mathematical datasets containing diverse notation. Experiments show that models trained on these datasets exhibit new SoTA performance on mathematical retrieval tasks. We publish our code, generated datasets, and pretrained mathematical models: https://github.com/aieng-lab/math-mutator.