When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
For researchers and practitioners automating literary evaluation, this work shows that reasoning supervision can harm performance in fixed-format rubric-based generation, a counterintuitive finding for LLM fine-tuning.
The authors constructed a dataset of 263,911 long-form stories with TTCW-based creativity annotations and fine-tuned Qwen3 models (4B and 8B) for literary review generation. Non-reasoning fine-tuning outperformed reasoning supervision, achieving a best evaluation score of 0.6820, while reasoning-supervised models suffered from parse failures and irrelevant output.
Automatic evaluation of long-form literary writing remains challenging, as generic LLM-as-Judge approaches may not fully capture creativity-related dimensions such as originality and flexibility. Although the Torrance Test of Creative Writing (TTCW) provides a structured creativity framework, and prior work has demonstrated reference-based TTCW evaluation at the pairwise level, no large-scale dataset exists for long-form TTCW-based literary review generation. We address this gap by constructing a dataset of 263,911 long-form stories, each annotated with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we fine-tune Qwen3 models at two scales, 4B and 8B, under two conditions: with and without reasoning content. Results show that non-reasoning fine-tuning achieves stronger and more stable performance, with the best setting reaching an evaluation score of 0.6820. Further analysis shows that reasoning-supervised models are more prone to parse failures, often continuing with irrelevant or repetitive reasoning-style text rather than completing the required 14-metric review report. These results suggest that, for fixed-format rubric-based review generation, reasoning supervision is not straightforwardly beneficial, and precise metric-aligned scoring remains challenging even after task-specific fine-tuning.