On The Direct Maximization of Quadratic Weighted Kappa
This work addresses a gap in understanding and optimizing a key metric used in machine learning contests, but it is incremental as it builds on existing metric definitions.
The paper tackled the problem of optimizing quadratic weighted kappa, a popular evaluation metric for integer rating predictions, by deriving a simpler definition and using it to address analytical optimization and mathematical properties.
In recent years, quadratic weighted kappa has been growing in popularity in the machine learning community as an evaluation metric in domains where the target labels to be predicted are drawn from integer ratings, usually obtained from human experts. For example, it was the metric of choice in several recent, high profile machine learning contests hosted on Kaggle : https://www.kaggle.com/c/asap-aes , https://www.kaggle.com/c/asap-sas , https://www.kaggle.com/c/diabetic-retinopathy-detection . Yet, little is understood about the nature of this metric, its underlying mathematical properties, where it fits among other common evaluation metrics such as mean squared error (MSE) and correlation, or if it can be optimized analytically, and if so, how. Much of this is due to the cumbersome way that this metric is commonly defined. In this paper we first derive an equivalent but much simpler, and more useful, definition for quadratic weighted kappa, and then employ this alternate form to address the above issues.