A Comprehensive Review on Deep Supervision: Theories and Applications
This is an incremental review paper that synthesizes existing knowledge on deep supervision for researchers in computer vision and deep learning.
The paper provides a comprehensive review of deep supervision techniques in neural networks, focusing on their theories and applications in computer vision, and proposes a new classification system for these networks.
Deep supervision, or known as 'intermediate supervision' or 'auxiliary supervision', is to add supervision at hidden layers of a neural network. This technique has been increasingly applied in deep neural network learning systems for various computer vision applications recently. There is a consensus that deep supervision helps improve neural network performance by alleviating the gradient vanishing problem, as one of the many strengths of deep supervision. Besides, in different computer vision applications, deep supervision can be applied in different ways. How to make the most use of deep supervision to improve network performance in different applications has not been thoroughly investigated. In this paper, we provide a comprehensive in-depth review of deep supervision in both theories and applications. We propose a new classification of different deep supervision networks, and discuss advantages and limitations of current deep supervision networks in computer vision applications.