Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets
This work addresses the problem of data scarcity in machine learning for domain experts, offering an incremental improvement by integrating user modeling into knowledge elicitation.
The paper tackles the challenge of making accurate predictions with small datasets by developing an interactive visualization approach to elicit tacit prior knowledge from domain experts about feature relevance, which significantly improves prediction accuracy in a controlled study on predicting scientific citation counts.
Providing accurate predictions is challenging for machine learning algorithms when the number of features is larger than the number of samples in the data. Prior knowledge can improve machine learning models by indicating relevant variables and parameter values. Yet, this prior knowledge is often tacit and only available from domain experts. We present a novel approach that uses interactive visualization to elicit the tacit prior knowledge and uses it to improve the accuracy of prediction models. The main component of our approach is a user model that models the domain expert's knowledge of the relevance of different features for a prediction task. In particular, based on the expert's earlier input, the user model guides the selection of the features on which to elicit user's knowledge next. The results of a controlled user study show that the user model significantly improves prior knowledge elicitation and prediction accuracy, when predicting the relative citation counts of scientific documents in a specific domain.