SteerEval: Inference-time Interventions Strengthen Multilingual Generalization in Neural Summarization Metrics
This addresses the bottleneck of robust multilingual evaluation for summarization metrics, which is incremental as it builds on known issues with pivot language misalignment.
The paper tackled the problem of inaccurate multilingual evaluation metrics for summarization by testing whether steering neural metrics' activations toward an English pivot improves correlation with human judgments, finding that test-time interventions increased effectiveness across diverse languages.
An increasing body of work has leveraged multilingual language models for Natural Language Generation tasks such as summarization. A major empirical bottleneck in this area is the shortage of accurate and robust evaluation metrics for many languages, which hinders progress. Recent studies suggest that multilingual language models often use English as an internal pivot language, and that misalignment with this pivot can lead to degraded downstream performance. Motivated by the hypothesis that this mismatch could also apply to multilingual neural metrics, we ask whether steering their activations toward an English pivot can improve correlation with human judgments. We experiment with encoder- and decoder-based metrics and find that test-time intervention methods are effective across the board, increasing metric effectiveness for diverse languages.