Using Experts' Opinions in Machine Learning Tasks
This addresses the challenge of improving prediction stability in sports analytics by incorporating expert insights, though it is incremental as it builds on existing competition tasks.
The paper tackles the problem of underutilizing experts' opinions in machine learning predictions by proposing a general framework and applying it to NCAA Men's Basketball game prediction, achieving lower log loss (best at 0.489) compared to top 2019 competition solutions (>0.503) and reaching top leaderboard positions.
In machine learning tasks, especially in the tasks of prediction, scientists tend to rely solely on available historical data and disregard unproven insights, such as experts' opinions, polls, and betting odds. In this paper, we propose a general three-step framework for utilizing experts' insights in machine learning tasks and build four concrete models for a sports game prediction case study. For the case study, we have chosen the task of predicting NCAA Men's Basketball games, which has been the focus of a group of Kaggle competitions in recent years. Results highly suggest that the good performance and high scores of the past models are a result of chance, and not because of a good-performing and stable model. Furthermore, our proposed models can achieve more steady results with lower log loss average (best at 0.489) compared to the top solutions of the 2019 competition (>0.503), and reach the top 1%, 10% and 1% in the 2017, 2018 and 2019 leaderboards, respectively.