A simple theory for training response of deep neural networks
This work addresses a theoretical gap in machine learning by providing insights into training dynamics and network fragility, but it is incremental as it builds on existing findings without introducing a new paradigm.
The paper tackles the problem of understanding training dynamics in deep neural networks, particularly the gaps between known behaviors like constant or power-law aging and complex phenomena such as network fragility, by analyzing a simple network model and showing that training response depends on factors like training stages, activation functions, and methods, with feature space reduction identified as a cause of fragility.
Deep neural networks give us a powerful method to model the training dataset's relationship between input and output. We can regard that as a complex adaptive system consisting of many artificial neurons that work as an adaptive memory as a whole. The network's behavior is training dynamics with a feedback loop from the evaluation of the loss function. We already know the training response can be constant or shows power law-like aging in some ideal situations. However, we still have gaps between those findings and other complex phenomena, like network fragility. To fill the gap, we introduce a very simple network and analyze it. We show the training response consists of some different factors based on training stages, activation functions, or training methods. In addition, we show feature space reduction as an effect of stochastic training dynamics, which can result in network fragility. Finally, we discuss some complex phenomena of deep networks.