DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation
This addresses a gap in benchmarking for real-world document processing, though it's incremental as it focuses on creating a new dataset rather than developing new methods.
The authors tackled the problem that existing benchmarks for document parsing and translation don't adequately represent real-world photographed documents with distortions, by introducing DocPTBench with over 1,300 photographed documents. Their experiments showed that transitioning from digital-born to photographed documents causes substantial performance declines: popular MLLMs dropped 18% in parsing and 12% in translation accuracy, while specialized models dropped 25%.
The advent of Multimodal Large Language Models (MLLMs) has unlocked the potential for end-to-end document parsing and translation. However, prevailing benchmarks such as OmniDocBench and DITrans are dominated by pristine scanned or digital-born documents, and thus fail to adequately represent the intricate challenges of real-world capture conditions, such as geometric distortions and photometric variations. To fill this gap, we introduce DocPTBench, a comprehensive benchmark specifically designed for Photographed Document Parsing and Translation. DocPTBench comprises over 1,300 high-resolution photographed documents from multiple domains, includes eight translation scenarios, and provides meticulously human-verified annotations for both parsing and translation. Our experiments demonstrate that transitioning from digital-born to photographed documents results in a substantial performance decline: popular MLLMs exhibit an average accuracy drop of 18% in end-to-end parsing and 12% in translation, while specialized document parsing models show significant average decrease of 25%. This substantial performance gap underscores the unique challenges posed by documents captured in real-world conditions and reveals the limited robustness of existing models. Dataset and code are available at https://github.com/Topdu/DocPTBench.