Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
For users of LLM-based writing/review tools, this addresses the problem of eliciting tacit expert criteria from inline comments, though the approach is incremental.
The paper proposes a method to learn reusable natural-language rubrics from inline comments on drafts, enabling comment prediction, rubric understanding, and automatic revision. Evaluations in real-world and controlled settings demonstrate the feasibility of distilling inline comments into rubrics.
Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision.