From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
For researchers and practitioners working on LLM-based writing tasks, this work provides a more precise evaluation and training methodology, though it is an incremental improvement over existing coarse-grained approaches.
The paper addresses the lack of fine-grained evaluation and reward modeling for writing-centric generation tasks in LLMs. They propose WEval, a fine-grained evaluation pipeline, and WRL, a reinforcement learning training framework, achieving substantial improvements across writing benchmarks.
Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.