Dialect-robust Evaluation of Generated Text
This addresses a fairness issue in NLG evaluation for users of diverse dialects, though it is incremental as it builds on existing metrics.
The paper tackles the problem of evaluating natural language generation (NLG) systems across dialects, showing that current metrics are not robust to dialect variation and can penalize lower-resource dialects. It introduces a suite of methods to assess dialect robustness and proposes NANO, a training schema that improves dialect robustness while enhancing performance on standard benchmarks.
Evaluation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. However, currently, there exists no way to quantify how metrics respond to change in the dialect of a generated utterance. We thus formalize dialect robustness and dialect awareness as goals for NLG evaluation metrics. We introduce a suite of methods and corresponding statistical tests one can use to assess metrics in light of the two goals. Applying the suite to current state-of-the-art metrics, we demonstrate that they are not dialect-robust and that semantic perturbations frequently lead to smaller decreases in a metric than the introduction of dialect features. As a first step to overcome this limitation, we propose a training schema, NANO, which introduces regional and language information to the pretraining process of a metric. We demonstrate that NANO provides a size-efficient way for models to improve the dialect robustness while simultaneously improving their performance on the standard metric benchmark.