Evaluating historical text normalization systems: How well do they generalize?
This work addresses evaluation challenges for researchers and practitioners in historical NLP, but it is incremental as it focuses on improving assessment methods rather than introducing new models.
The paper tackled the problem of evaluating historical text normalization systems by identifying issues that hinder practical assessment, and found that while neural models generalize well to unseen words across five languages, they offer no clear advantage over a naive baseline for downstream POS tagging in an English historical collection.
We highlight several issues in the evaluation of historical text normalization systems that make it hard to tell how well these systems would actually work in practice---i.e., for new datasets or languages; in comparison to more naïve systems; or as a preprocessing step for downstream NLP tools. We illustrate these issues and exemplify our proposed evaluation practices by comparing two neural models against a naïve baseline system. We show that the neural models generalize well to unseen words in tests on five languages; nevertheless, they provide no clear benefit over the naïve baseline for downstream POS tagging of an English historical collection. We conclude that future work should include more rigorous evaluation, including both intrinsic and extrinsic measures where possible.