Information Theory and its Relation to Machine Learning
It addresses the foundational problem of learning target selection in machine learning, but is incremental as it builds on existing information theory studies.
The paper proposes a new perspective on machine learning organized into four basic problems, focusing on 'What to learn?' or learning target selection, and reviews connections between information theoretical learning and machine learning, including a theorem on similarity and information measures and a conjecture for a unified interpretation.
In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the first level of "What to learn?", or "Learning Target Selection". Towards this primary problem within the four levels, I briefly review the existing studies about the connection between information theoretical learning (ITL [1]) and machine learning. A theorem is given on the relation between the empirically-defined similarity measure and information measures. Finally, a conjecture is proposed for pursuing a unified mathematical interpretation to learning target selection.