PopEval: A Character-Level Approach to End-To-End Evaluation Compatible with Word-Level Benchmark Dataset
This work addresses the need for updated evaluation metrics in OCR for natural scene applications, offering an incremental improvement over traditional methods.
The authors tackled the mismatch between existing OCR evaluation methods and modern scene text recognition by proposing PopEval, a character-level evaluation approach that aligns better with human qualitative assessments and is compatible with word-level benchmark datasets.
The most prevalent scope of interest for OCR applications used to be scanned documents, but it has now shifted towards the natural scene. Despite the change of times, the existing evaluation methods are still based on the old criteria suited better for the past interests. In this paper, we propose PopEval, a novel evaluation approach for the recent OCR interests. The new and past evaluation algorithms were compared through the results on various datasets and OCR models. Compared to the other evaluation methods, the proposed evaluation algorithm was closer to the human's qualitative evaluation than other existing methods. Although the evaluation algorithm was devised as a character-level approach, the comparative experiment revealed that PopEval is also compatible on existing benchmark datasets annotated at word-level. The proposed evaluation algorithm is not only applicable to current end-to-end tasks, but also suggests a new direction to redesign the evaluation concept for further OCR researches.