A Review of Feature Selection Methods Based on Mutual Information
This is an incremental review that synthesizes and clarifies existing methods for researchers in machine learning and data analysis.
The paper reviews information theoretic feature selection methods, defining key concepts like feature relevance and redundancy, and presents a unifying theoretical framework to analyze existing heuristic criteria.
In this work we present a review of the state of the art of information theoretic feature selection methods. The concepts of feature relevance, redundance and complementarity (synergy) are clearly defined, as well as Markov blanket. The problem of optimal feature selection is defined. A unifying theoretical framework is described, which can retrofit successful heuristic criteria, indicating the approximations made by each method. A number of open problems in the field are presented.