Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts
This work addresses the problem of inconsistent and biased human evaluations for NLG systems, offering a scalable alternative for researchers and practitioners, though it is incremental in automating prompt design.
The paper tackles the challenge of evaluating natural language generation systems by addressing the sensitivity of LLM-based evaluators to prompt design, proposing an inversion learning method that automatically generates model-specific evaluation prompts from a single sample, eliminating manual engineering and improving efficiency and robustness.
Evaluating natural language generation systems is challenging due to the diversity of valid outputs. While human evaluation is the gold standard, it suffers from inconsistencies, lack of standardisation, and demographic biases, limiting reproducibility. LLM-based evaluators offer a scalable alternative but are highly sensitive to prompt design, where small variations can lead to significant discrepancies. In this work, we propose an inversion learning method that learns effective reverse mappings from model outputs back to their input instructions, enabling the automatic generation of highly effective, model-specific evaluation prompts. Our method requires only a single evaluation sample and eliminates the need for time-consuming manual prompt engineering, thereby improving both efficiency and robustness. Our work contributes toward a new direction for more robust and efficient LLM-based evaluation.