An Empirical Analysis of Factual Errors in Human-Written Text and its Application
For researchers in NLP and fact-checking, this work highlights a gap in FED for human-written text and provides a taxonomy, but the results are preliminary.
The paper addresses the neglected problem of factual error detection (FED) in human-written text, proposing a taxonomy of human-induced errors from newspaper corrections. Evaluation shows that even GPT-5.4 achieves only 52% word-level F1 on synthetic data, highlighting task difficulty.
Factual Error Detection (FED), which is the task of identifying factually incorrect spans in a given text, has long been recognized as an important research problem. However, with the rapid rise of large language models (LLMs), research attention has shifted toward factual errors specific to LLM-generated text (hallucinations) and their detection. As a result, the detection of factual errors in human-written text has been relatively neglected. To address this gap, we first distill a taxonomy of human-induced factual errors by analyzing corrections of newspaper articles, a representative source of text that is guaranteed to be human-written and contains few grammatical errors. Our analysis revealed that there are characteristic categories such as kanji misconversions and numeral classifier errors, which are not focused in existing hallucination benchmarks. Based on the taxonomy, we then evaluate the FED capability of vanilla LLMs on synthesized realistic test cases and real corrections. Experimental results demonstrated that even high-performance LLMs such as GPT-5.4 achieved only word-level F1 score of 52% on the synthetic evaluation data, highlighting the task difficulty. Furthermore, a detailed analysis by detection difficulty revealed the current state of FED.