Unfair items detection in educational measurement
This addresses the problem of inconsistent fairness assessments in educational testing for measurement professionals, though it is incremental as it builds on existing fairness concepts.
The paper tackled the lack of consensus on defining item fairness in educational measurement by proposing a continuous measure of item unfairness, which identifies unfair items that conventional methods miss and aligns with expert judgments, as demonstrated on a real test.
Measurement professionals cannot come to an agreement on the definition of the term 'item fairness'. In this paper a continuous measure of item unfairness is proposed. The more the unfairness measure deviates from zero, the less fair the item is. If the measure exceeds the cutoff value, the item is identified as definitely unfair. The new approach can identify unfair items that would not be identified with conventional procedures. The results are in accord with experts' judgments on the item qualities. Since no assumptions about scores distributions and/or correlations are assumed, the method is applicable to any educational test. Its performance is illustrated through application to scores of a real test.