Automated Alignment of Math Items to Content Standards in Large-Scale Assessments Using Language Models
This addresses the need for accurate alignment to ensure valid score interpretation in assessments, but it is incremental as it compares existing methods without introducing a new paradigm.
The study tackled the problem of aligning math items to content standards in large-scale assessments by evaluating automated methods, with DeBERTa-v3-base achieving a weighted-average F1 score of 0.950 for domain alignment and RoBERTa-large achieving an F1 score of 0.869 for skill alignment.
Accurate alignment of items to content standards is critical for valid score interpretation in large-scale assessments. This study evaluates three automated paradigms for aligning items with four domain and nineteen skill labels. First, we extracted embeddings and trained multiple classical supervised machine learning models, and further investigated the impact of dimensionality reduction on model performance. Second, we fine-tuned eight BERT model and its variants for both domain and skill alignment. Third, we explored ensemble learning with majority voting and stacking with multiple meta-models. The DeBERTa-v3-base achieved the highest weighted-average F1 score of 0.950 for domain alignment while the RoBERTa-large yielded the highest F1 score of 0.869 for skill alignment. Ensemble models did not surpass the best-performing language models. Dimension reduction enhanced linear classifiers based on embeddings but did not perform better than language models. This study demonstrated different methods in automated item alignment to content standards.}