TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
This addresses a gap in evaluating LLMs for scientific documentation generation, but it is incremental as it focuses on a specific domain benchmark.
The authors tackled the lack of benchmarks for evaluating LaTeX code generation by LLMs by introducing TeXpert, a multi-level dataset, and found that LLMs perform poorly in LaTeX tasks with accuracy dropping as complexity increases, and open-source models rival closed-source ones.
LaTeX's precision and flexibility in typesetting have made it the gold standard for the preparation of scientific documentation. Large Language Models (LLMs) present a promising opportunity for researchers to produce publication-ready material using LaTeX with natural language instructions, yet current benchmarks completely lack evaluation of this ability. By introducing TeXpert, our benchmark dataset with natural language prompts for generating LaTeX code focused on components of scientific documents across multiple difficulty levels, we conduct an in-depth analysis of LLM performance in this regard and identify frequent error types. Our evaluation across open and closed-source LLMs highlights multiple key findings: LLMs excelling on standard benchmarks perform poorly in LaTeX generation with a significant accuracy drop-off as the complexity of tasks increases; open-source models like DeepSeek v3 and DeepSeek Coder strongly rival closed-source counterparts in LaTeX tasks; and formatting and package errors are unexpectedly prevalent, suggesting a lack of diverse LaTeX examples in the training datasets of most LLMs. Our dataset, code, and model evaluations are available at https://github.com/knowledge-verse-ai/TeXpert.