NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding
This addresses the problem of limited generalization in document AI for real-world note-taking, providing a new evaluation standard for researchers and practitioners in visual document understanding.
The paper tackles the challenge of understanding and reasoning over academic handwritten notes, which include mathematical equations and diagrams, by introducing NoTeS-Bank, a benchmark for neural transcription and search in note-based question answering, and it benchmarks state-of-the-art models, exposing limitations in structured transcription and reasoning.
Understanding and reasoning over academic handwritten notes remains a challenge in document AI, particularly for mathematical equations, diagrams, and scientific notations. Existing visual question answering (VQA) benchmarks focus on printed or structured handwritten text, limiting generalization to real-world note-taking. To address this, we introduce NoTeS-Bank, an evaluation benchmark for Neural Transcription and Search in note-based question answering. NoTeS-Bank comprises complex notes across multiple domains, requiring models to process unstructured and multimodal content. The benchmark defines two tasks: (1) Evidence-Based VQA, where models retrieve localized answers with bounding-box evidence, and (2) Open-Domain VQA, where models classify the domain before retrieving relevant documents and answers. Unlike classical Document VQA datasets relying on optical character recognition (OCR) and structured data, NoTeS-BANK demands vision-language fusion, retrieval, and multimodal reasoning. We benchmark state-of-the-art Vision-Language Models (VLMs) and retrieval frameworks, exposing structured transcription and reasoning limitations. NoTeS-Bank provides a rigorous evaluation with NDCG@5, MRR, Recall@K, IoU, and ANLS, establishing a new standard for visual document understanding and reasoning.