A Reproducible, Scalable Pipeline for Synthesizing Autoregressive Model Literature
This tool addresses the scalability problem for researchers and practitioners overwhelmed by the volume of autoregressive model papers, though it is incremental as it builds on existing automation and reproducibility techniques.
The authors tackled the challenge of manually surveying and reproducing the growing literature on autoregressive generative models by developing an open-source, reproducible pipeline that automates document retrieval, metadata extraction, and experiment reproduction, achieving F1 scores above 0.85 on key tasks and enabling faithful reproduction with test perplexities within 1-3% of original reports.
The accelerating pace of research on autoregressive generative models has produced thousands of papers, making manual literature surveys and reproduction studies increasingly impractical. We present a fully open-source, reproducible pipeline that automatically retrieves candidate documents from public repositories, filters them for relevance, extracts metadata, hyper-parameters and reported results, clusters topics, produces retrieval-augmented summaries and generates containerised scripts for re-running selected experiments. Quantitative evaluation on 50 manually-annotated papers shows F1 scores above 0.85 for relevance classification, hyper-parameter extraction and citation identification. Experiments on corpora of up to 1000 papers demonstrate near-linear scalability with eight CPU workers. Three case studies -- AWD-LSTM on WikiText-2, Transformer-XL on WikiText-103 and an autoregressive music model on the Lakh MIDI dataset -- confirm that the extracted settings support faithful reproduction, achieving test perplexities within 1--3% of the original reports.