Measuring Model Complexity of Neural Networks with Curve Activation Functions
This work addresses a foundational gap in understanding model complexity for neural networks with curve activation functions, which is incremental as it extends existing methods from piecewise linear to curve functions.
The paper tackles the problem of measuring model complexity for neural networks with curve activation functions, which was previously an open issue, by proposing a piecewise linear approximation framework (LANN) and deriving a complexity measure based on the upper bound of linear regions, with experimental results showing that overfitting correlates positively with increased complexity and that regularizations suppress this increase.
It is fundamental to measure model complexity of deep neural networks. The existing literature on model complexity mainly focuses on neural networks with piecewise linear activation functions. Model complexity of neural networks with general curve activation functions remains an open problem. To tackle the challenge, in this paper, we first propose the linear approximation neural network (LANN for short), a piecewise linear framework to approximate a given deep model with curve activation function. LANN constructs individual piecewise linear approximation for the activation function of each neuron, and minimizes the number of linear regions to satisfy a required approximation degree. Then, we analyze the upper bound of the number of linear regions formed by LANNs, and derive the complexity measure based on the upper bound. To examine the usefulness of the complexity measure, we experimentally explore the training process of neural networks and detect overfitting. Our results demonstrate that the occurrence of overfitting is positively correlated with the increase of model complexity during training. We find that the $L^1$ and $L^2$ regularizations suppress the increase of model complexity. Finally, we propose two approaches to prevent overfitting by directly constraining model complexity, namely neuron pruning and customized $L^1$ regularization.