CRACQ: A Multi-Dimensional Approach To Automated Document Assessment
This addresses the need for interpretable and trait-specific evaluation of machine-generated text, though it is incremental in building on existing AES methods.
The paper tackles the problem of automated document assessment by introducing CRACQ, a multi-dimensional framework that evaluates documents across five traits, and shows it produces more stable and interpretable judgments than LLM-based evaluation in preliminary tests.
This paper presents CRACQ, a multi-dimensional evaluation framework tailored to evaluate documents across f i v e specific traits: Coherence, Rigor, Appropriateness, Completeness, and Quality. Building on insights from traitbased Automated Essay Scoring (AES), CRACQ expands its fo-cus beyond essays to encompass diverse forms of machine-generated text, providing a rubricdriven and interpretable methodology for automated evaluation. Unlike singlescore approaches, CRACQ integrates linguistic, semantic, and structural signals into a cumulative assessment, enabling both holistic and trait-level analysis. Trained on 500 synthetic grant pro-posals, CRACQ was benchmarked against an LLM-as-a-judge and further tested on both strong and weak real applications. Preliminary results in-dicate that CRACQ produces more stable and interpretable trait-level judgments than direct LLM evaluation, though challenges in reliability and domain scope remain