'Almost Sure' Chaotic Properties of Machine Learning Methods
This work provides a theoretical foundation for chaotic dynamics in machine learning, which is foundational for the entire field of ML/AI.
The paper tackles the problem of understanding chaotic behavior in machine learning methods by establishing that iterative learning processes are 'almost surely' chaotic, explaining counterintuitive properties observed in deep learning.
It has been demonstrated earlier that universal computation is 'almost surely' chaotic. Machine learning is a form of computational fixed point iteration, iterating over the computable function space. We showcase some properties of this iteration, and establish in general that the iteration is 'almost surely' of chaotic nature. This theory explains the observation in the counter intuitive properties of deep learning methods. This paper demonstrates that these properties are going to be universal to any learning method.