Cohort-attention Evaluation Metric against Tied Data: Studying Performance of Classification Models in Cancer Detection
This addresses evaluation challenges for AI models in medical screening, particularly cancer detection, but appears incremental as it builds on existing metrics with modifications for fairness and interpretability.
The paper tackles the problem of biased evaluations in AI-driven cancer detection due to imbalanced data and cohort variations by proposing the Cohort-Attention Evaluation Metrics (CAT) framework, which introduces patient-level assessment and entropy-based weighting to enhance reliability and fairness.
Artificial intelligence (AI) has significantly improved medical screening accuracy, particularly in cancer detection and risk assessment. However, traditional classification metrics often fail to account for imbalanced data, varying performance across cohorts, and patient-level inconsistencies, leading to biased evaluations. We propose the Cohort-Attention Evaluation Metrics (CAT) framework to address these challenges. CAT introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity. Key metrics like CATSensitivity (CATSen), CATSpecificity (CATSpe), and CATMean ensure balanced and fair evaluation across diverse populations. This approach enhances predictive reliability, fairness, and interpretability, providing a robust evaluation method for AI-driven medical screening models.