Human Uncertainty and Ranking Error -- The Secret of Successful Evaluation in Predictive Data Mining
This addresses the issue of unreliable evaluations in predictive data mining, particularly for recommender systems, by highlighting and mitigating human-induced errors, though it is incremental in nature.
The paper tackles the problem of human uncertainty affecting comparative evaluations in data mining, revealing biases in prediction metrics and error probabilities in algorithm rankings, and provides mathematical proofs and solution strategies.
One of the most crucial issues in data mining is to model human behaviour in order to provide personalisation, adaptation and recommendation. This usually involves implicit or explicit knowledge, either by observing user interactions, or by asking users directly. But these sources of information are always subject to the volatility of human decisions, making utilised data uncertain to a particular extent. In this contribution, we elaborate on the impact of this human uncertainty when it comes to comparative assessments of different data mining approaches. In particular, we reveal two problems: (1) biasing effects on various metrics of model-based prediction and (2) the propagation of uncertainty and its thus induced error probabilities for algorithm rankings. For this purpose, we introduce a probabilistic view and prove the existence of those problems mathematically, as well as provide possible solution strategies. We exemplify our theory mainly in the context of recommender systems along with the metric RMSE as a prominent example of precision quality measures.