The principles of adaptation in organisms and machines I: machine learning, information theory, and thermodynamics
This work provides a foundational perspective on adaptation mechanisms, potentially impacting all of ML/AI by bridging theoretical frameworks, but it is incremental as it reviews and synthesizes existing hypotheses without introducing new empirical results.
The paper tackles the problem of understanding how organisms adapt to their environment by integrating machine learning, information theory, and thermodynamics, resulting in a unified view that links causal learning to the second law of thermodynamics and inference dynamics to thermodynamic processes.
How do organisms recognize their environment by acquiring knowledge about the world, and what actions do they take based on this knowledge? This article examines hypotheses about organisms' adaptation to the environment from machine learning, information-theoretic, and thermodynamic perspectives. We start with constructing a hierarchical model of the world as an internal model in the brain, and review standard machine learning methods to infer causes by approximately learning the model under the maximum likelihood principle. This in turn provides an overview of the free energy principle for an organism, a hypothesis to explain perception and action from the principle of least surprise. Treating this statistical learning as communication between the world and brain, learning is interpreted as a process to maximize information about the world. We investigate how the classical theories of perception such as the infomax principle relates to learning the hierarchical model. We then present an approach to the recognition and learning based on thermodynamics, showing that adaptation by causal learning results in the second law of thermodynamics whereas inference dynamics that fuses observation with prior knowledge forms a thermodynamic process. These provide a unified view on the adaptation of organisms to the environment.