From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering
For practitioners building RAG systems, this work provides the first systematic comparison of PDF processing pipelines, showing that preprocessing choices (especially hierarchy-aware chunking and metadata enrichment) matter more than the conversion tool itself.
The study evaluates four PDF-to-Markdown conversion frameworks for RAG systems, finding that Docling with hierarchical splitting and image descriptions achieves 94.1% accuracy on a Portuguese administrative document benchmark, outperforming naive PDFLoader (86.9%) and approaching manually curated Markdown (97.1%). Data preparation quality is identified as the dominant factor in RAG performance.
Retrieval-Augmented Generation (RAG) systems depend critically on the quality of document preprocessing, yet no prior study has evaluated PDF processing frameworks by their impact on downstream question-answering accuracy. We address this gap through a systematic comparison of four open-source PDF-to-Markdown conversion frameworks, Docling, MinerU, Marker, and DeepSeek OCR, across 19 pipeline configurations for extracting text and other contents from PDFs, varying the conversion tool, cleaning transformations, splitting strategy, and metadata enrichment. Evaluation was performed using a manually curated 50-question benchmark over a corpus of 36 Portuguese administrative documents (1,706 pages, ~492K words), with LLM-as-judge scoring averaged over 10 runs. Two baselines bounded the results: naïve PDFLoader (86.9%) and manually curated Markdown (97.1%). Docling with hierarchical splitting and image descriptions achieved the highest automated accuracy (94.1%). Metadata enrichment and hierarchy-aware chunking contributed more to accuracy than the conversion framework choice alone. Font-based hierarchy rebuilding consistently outperformed LLM-based approaches. An exploratory GraphRAG implementation scored only 82%, underperforming basic RAG, suggesting that naïve knowledge graph construction without ontological guidance does not yet justify its added complexity. These findings demonstrate that data preparation quality is the dominant factor in RAG system performance.