The Interaction of Entropy-Based Discretization and Sample Size: An Empirical Study
This addresses a methodological issue in data mining for researchers and practitioners, revealing biases in discretization practices that could affect model evaluation, though it is incremental as it builds on prior suggestions without empirical evidence.
The study investigated how sample size interacts with entropy-based discretization (CAIM) and found that using discretization within cross-validation folds introduces significant optimistic bias in performance metrics, especially at smaller sample sizes, with over 117,000 models tested on seven UCI datasets.
An empirical investigation of the interaction of sample size and discretization - in this case the entropy-based method CAIM (Class-Attribute Interdependence Maximization) - was undertaken to evaluate the impact and potential bias introduced into data mining performance metrics due to variation in sample size as it impacts the discretization process. Of particular interest was the effect of discretizing within cross-validation folds averse to outside discretization folds. Previous publications have suggested that discretizing externally can bias performance results; however, a thorough review of the literature found no empirical evidence to support such an assertion. This investigation involved construction of over 117,000 models on seven distinct datasets from the UCI (University of California-Irvine) Machine Learning Library and multiple modeling methods across a variety of configurations of sample size and discretization, with each unique "setup" being independently replicated ten times. The analysis revealed a significant optimistic bias as sample sizes decreased and discretization was employed. The study also revealed that there may be a relationship between the interaction that produces such bias and the numbers and types of predictor attributes, extending the "curse of dimensionality" concept from feature selection into the discretization realm. Directions for further exploration are laid out, as well some general guidelines about the proper application of discretization in light of these results.