Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation
This addresses the problem of scalable training for LLMs in creative and ethical domains without ground truth, though it appears incremental as it builds on existing self-rewarding approaches.
The paper tackles the challenge of training large language models for non-verifiable tasks like creative writing by addressing limitations in LLM-as-Judge methods, introducing CoNL, a framework that uses multi-agent self-play for meta-evaluation, and achieves consistent improvements over baselines on five benchmarks.
Training large language models (LLMs) for non-verifiable tasks, such as creative writing, dialogue, and ethical reasoning, remains challenging due to the absence of ground-truth labels. While LLM-as-Judge approaches offer a scalable alternative to human feedback, they face a fundamental limitation: performance is constrained by the evaluator's own quality. If the judge cannot recognize good solutions, it cannot provide useful training signals, and evaluation biases (e.g., favoring verbosity over quality) remain unaddressed. This motivates meta-evaluation: the ability to evaluate and improve the evaluator itself. We introduce CoNL, a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play. Our key insight: critique quality can be measured by whether it helps others improve their solutions. In CoNL, multiple agents sharing the same policy engage in structured conversations to propose, critique, and revise solutions. Critiques that enable solution improvements earn a diagnostic reward, creating explicit supervision for meta-evaluation and enabling joint optimization of generation and judging capabilities through self-play, without external judges or ground truth. Experiments on five benchmarks show that CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.