Context-Driven Data Mining through Bias Removal and Data Incompleteness Mitigation
This addresses data mining challenges for practitioners by introducing a new lifecycle approach, though it appears incremental as it builds on existing data science frameworks.
The paper tackles data quality issues like bias and incompleteness in data mining by proposing a Context-driven Data Science Lifecycle (C-DSL), with results evaluated using metrics such as R2 value and confusion matrices in sports event case studies.
The results of data mining endeavors are majorly driven by data quality. Throughout these deployments, serious show-stopper problems are still unresolved, such as: data collection ambiguities, data imbalance, hidden biases in data, the lack of domain information, and data incompleteness. This paper is based on the premise that context can aid in mitigating these issues. In a traditional data science lifecycle, context is not considered. Context-driven Data Science Lifecycle (C-DSL); the main contribution of this paper, is developed to address these challenges. Two case studies (using data-sets from sports events) are developed to test C-DSL. Results from both case studies are evaluated using common data mining metrics such as: coefficient of determination (R2 value) and confusion matrices. The work presented in this paper aims to re-define the lifecycle and introduce tangible improvements to its outcomes.