E-Gen: Leveraging E-Graphs to Improve Continuous Representations of Symbolic Expressions
This work addresses data constraints in mathematical expression embeddings for NLP applications, offering an incremental improvement through novel dataset generation.
The paper tackled the problem of limited training data for embedding mathematical expressions by introducing E-Gen, an e-graph-based dataset generation scheme that creates large and diverse datasets, leading to embedding models that outperform state-of-the-art large language models on mathematical language processing tasks.
Vector representations have been pivotal in advancing natural language processing (NLP), with prior research focusing on embedding techniques for mathematical expressions using mathematically equivalent formulations. While effective, these approaches are constrained by the size and diversity of training data. In this work, we address these limitations by introducing E-Gen, a novel e-graph-based dataset generation scheme that synthesizes large and diverse mathematical expression datasets, surpassing prior methods in size and operator variety. Leveraging this dataset, we train embedding models using two strategies: (1) generating mathematically equivalent expressions, and (2) contrastive learning to explicitly group equivalent expressions. We evaluate these embeddings on both in-distribution and out-of-distribution mathematical language processing tasks, comparing them against prior methods. Finally, we demonstrate that our embedding-based approach outperforms state-of-the-art large language models (LLMs) on several tasks, underscoring the necessity of optimizing embedding methods for the mathematical data modality. The source code and datasets are available at https://github.com/MLPgroup/E-Gen.