Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Papers
This addresses the time-consuming and subjective process of diagram generation for researchers and developers, establishing a foundational benchmark for reproducible research in automated scientific visualization.
The paper tackles the problem of manually creating system architecture diagrams from scientific papers by introducing a novel benchmark with 3,000 paper-diagram pairs and a three-tiered evaluation metric, and proposes Paper2SysArch, an end-to-end system that achieves a composite score of 69.0 on a challenging subset.
The manual creation of system architecture diagrams for scientific papers is a time-consuming and subjective process, while existing generative models lack the necessary structural control and semantic understanding for this task. A primary obstacle hindering research and development in this domain has been the profound lack of a standardized benchmark to quantitatively evaluate the automated generation of diagrams from text. To address this critical gap, we introduce a novel and comprehensive benchmark, the first of its kind, designed to catalyze progress in automated scientific visualization. It consists of 3,000 research papers paired with their corresponding high-quality ground-truth diagrams and is accompanied by a three-tiered evaluation metric assessing semantic accuracy, layout coherence, and visual quality. Furthermore, to establish a strong baseline on this new benchmark, we propose Paper2SysArch, an end-to-end system that leverages multi-agent collaboration to convert papers into structured, editable diagrams. To validate its performance on complex cases, the system was evaluated on a manually curated and more challenging subset of these papers, where it achieves a composite score of 69.0. This work's principal contribution is the establishment of a large-scale, foundational benchmark to enable reproducible research and fair comparison. Meanwhile, our proposed system serves as a viable proof-of-concept, demonstrating a promising path forward for this complex task.