Think Global, Act Local: Relating DNN generalisation and node-level SNR
This work addresses the open problem of understanding DNN generalization for researchers in machine learning, offering insights into training and regularization, but it appears incremental as it builds on existing information theory principles without introducing a new paradigm.
The paper investigates the relationship between node-level Signal-to-Noise Ratio (SNR) in deep neural networks and generalization performance, finding that weight sets promoting good SNR also exhibit improved generalization, with examples provided to support this connection.
The reasons behind good DNN generalisation remain an open question. In this paper we explore the problem by looking at the Signal-to-Noise Ratio of nodes in the network. Starting from information theory principles, it is possible to derive an expression for the SNR of a DNN node output. Using this expression we construct figures-of-merit that quantify how well the weights of a node optimise SNR (or, equivalently, information rate). Applying these figures-of-merit, we give examples indicating that weight sets that promote good SNR performance also exhibit good generalisation. In addition, we are able to identify the qualities of weight sets that exhibit good SNR behaviour and hence promote good generalisation. This leads to a discussion of how these results relate to network training and regularisation. Finally, we identify some ways that these observations can be used in training design.