From Compass and Ruler to Convolution and Nonlinearity: On the Surprising Difficulty of Understanding a Simple CNN Solving a Simple Geometric Estimation Task
This addresses the challenge of interpretability in neural networks, which is critical for their use in safety-sensitive systems, though it is incremental as it focuses on a simplified, well-posed task.
The researchers tackled the problem of interpreting a simple convolutional neural network (CNN) trained to estimate the radius of a centered pulse in 1D signals or disk in 2D images, finding that understanding the learned model is surprisingly difficult and counter-intuitive, but they achieved a complete theoretical analysis in the 1D case to explain the role of architecture, filters, and activation functions.
Neural networks are omnipresent, but remain poorly understood. Their increasing complexity and use in critical systems raises the important challenge to full interpretability. We propose to address a simple well-posed learning problem: estimating the radius of a centred pulse in a one-dimensional signal or of a centred disk in two-dimensional images using a simple convolutional neural network. Surprisingly, understanding what trained networks have learned is difficult and, to some extent, counter-intuitive. However, an in-depth theoretical analysis in the one-dimensional case allows us to comprehend constraints due to the chosen architecture, the role of each filter and of the nonlinear activation function, and every single value taken by the weights of the model. Two fundamental concepts of neural networks arise: the importance of invariance and of the shape of the nonlinear activation functions.