Reference-less Measure of Faithfulness for Grammatical Error Correction
This addresses the need for better evaluation metrics in GEC by providing a semantic complement to existing grammaticality measures, though it is incremental as it builds on prior reference-less approaches.
The paper tackles the problem of measuring semantic faithfulness in Grammatical Error Correction (GEC) by proposing USim, a reference-less measure that compares semantic symbolic structures between source and correction, with experiments showing it consistently scores valid corrections higher and invalid ones lower.
We propose USim, a semantic measure for Grammatical Error Correction (GEC) that measures the semantic faithfulness of the output to the source, thereby complementing existing reference-less measures (RLMs) for measuring the output's grammaticality. USim operates by comparing the semantic symbolic structure of the source and the correction, without relying on manually-curated references. Our experiments establish the validity of USim, by showing that (1) semantic annotation can be consistently applied to ungrammatical text; (2) valid corrections obtain a high USim similarity score to the source; and (3) invalid corrections obtain a lower score.