The Case for Meta-Cognitive Machine Learning: On Model Entropy and Concept Formation in Deep Learning
This work proposes a foundational shift in defining machine learning to include internal objectives, potentially impacting all of ML/AI by encouraging more holistic and efficient learning approaches.
The paper argues that machine learning should incorporate meta-cognitive strategies to reason over its own learning process, proposing a model entropy function to quantify internal learning efficiency and suggesting that minimizing it leads to concept formation.
Machine learning is usually defined in behaviourist terms, where external validation is the primary mechanism of learning. In this paper, I argue for a more holistic interpretation in which finding more probable, efficient and abstract representations is as central to learning as performance. In other words, machine learning should be extended with strategies to reason over its own learning process, leading to so-called meta-cognitive machine learning. As such, the de facto definition of machine learning should be reformulated in these intrinsically multi-objective terms, taking into account not only the task performance but also internal learning objectives. To this end, we suggest a "model entropy function" to be defined that quantifies the efficiency of the internal learning processes. It is conjured that the minimization of this model entropy leads to concept formation. Besides philosophical aspects, some initial illustrations are included to support the claims.