On Evaluating the Quality of Rule-Based Classification Systems
This work addresses a methodological gap for researchers and practitioners in machine learning, but it is incremental as it critiques existing metrics without proposing new ones.
The paper tackles the problem of evaluating rule-based classification systems by arguing that traditional indicators like predictive accuracy and coverage are insufficient, as systems with good scores can still be trivially improved, and it illustrates this with theoretical examples.
Two indicators are classically used to evaluate the quality of rule-based classification systems: predictive accuracy, i.e. the system's ability to successfully reproduce learning data and coverage, i.e. the proportion of possible cases for which the logical rules constituting the system apply. In this work, we claim that these two indicators may be insufficient, and additional measures of quality may need to be developed. We theoretically show that classification systems presenting "good" predictive accuracy and coverage can, nonetheless, be trivially improved and illustrate this proposition with examples.