Computing Linear Regions in Neural Networks with Skip Connections
This work addresses the challenge of understanding and training neural networks, particularly for researchers in machine learning, but it is incremental as it builds on existing tropical geometry methods.
The paper tackled the problem of analyzing neural networks with skip connections by computing linear regions using tropical geometry, and found that skip connections help mitigate overfitting and improve training difficulty.
Neural networks are important tools in machine learning. Representing piecewise linear activation functions with tropical arithmetic enables the application of tropical geometry. Algorithms are presented to compute regions where the neural networks are linear maps. Through computational experiments, we provide insights on the difficulty to train neural networks, in particular on the problems of overfitting and on the benefits of skip connections.