Evaluating Human and Machine Confidence in Phishing Email Detection: A Comparative Study
This research addresses the challenge of improving human-AI cooperation in phishing detection, offering insights for transparent AI systems, though it is incremental as it builds on existing comparative studies.
This study tackled the problem of distinguishing phishing emails from legitimate ones by comparing human cognition and machine learning models, finding that machines achieve good accuracy but with varying confidence, while humans use diverse linguistic cues and maintain more consistent confidence, with aging affecting detection but language proficiency having minimal impact.
Identifying deceptive content like phishing emails demands sophisticated cognitive processes that combine pattern recognition, confidence assessment, and contextual analysis. This research examines how human cognition and machine learning models work together to distinguish phishing emails from legitimate ones. We employed three interpretable algorithms Logistic Regression, Decision Trees, and Random Forests training them on both TF-IDF features and semantic embeddings, then compared their predictions against human evaluations that captured confidence ratings and linguistic observations. Our results show that machine learning models provide good accuracy rates, but their confidence levels vary significantly. Human evaluators, on the other hand, use a greater variety of language signs and retain more consistent confidence. We also found that while language proficiency has minimal effect on detection performance, aging does. These findings offer helpful direction for creating transparent AI systems that complement human cognitive functions, ultimately improving human-AI cooperation in challenging content analysis tasks.