How Complexity Contributes to Learning Opacity in Machine Learning
For ML researchers and practitioners, this work provides a theoretical explanation for why certain aspects of learning opacity may be unavoidable, challenging assumptions about transparency.
The paper argues that neural network learning opacity stems from dynamical complexity, identifying three irreducible properties (sensitivity to initialization, feedback in optimization, and data sensitivity) that make learning fundamentally opaque.
Machine learning (ML) algorithms are known to be opaque. We do not know the reasons for their predictions. The learning process leading to the prediction function is also opaque. We do not fully understand the time evolution of the weight values of neural nets (NN) and related dynamical phenomena. While prediction opacity is widely studied, learning opacity remains largely underexplored. This article studies learning opacity trough the lens of complex dynamical systems. We argue that NN learning is essentially a complex system and that learning opacity is due to dynamical complexity and the epistemological challenges that arise from it. We identify three key properties of training complexity -- sensitivity to weight initialization, feedback in gradient based optimization, and sensitivity to the training data -- and show how each contributes to learning opacity. As these properties are fundamental to the learning process damping or eliminating them would fundamentally alter how ML systems learn. Some sources of opacity in ML may hence be irreducible.