Making Sense of Hidden Layer Information in Deep Networks by Learning Hierarchical Targets
This work addresses the challenge of leveraging intermediate representations in deep learning for tasks requiring multi-level outputs, though it is incremental in nature.
The paper tackles the problem of extracting useful information from hidden layers in deep networks by introducing hierarchical target branches, achieving improved accuracy on a text classification task using the 20 Newsgroups dataset.
This paper proposes an architecture for deep neural networks with hidden layer branches that learn targets of lower hierarchy than final layer targets. The branches provide a channel for enforcing useful information in hidden layer which helps in attaining better accuracy, both for the final layer and hidden layers. The shared layers modify their weights using the gradients of all cost functions higher than the branching layer. This model provides a flexible inference system with many levels of targets which is modular and can be used efficiently in situations requiring different levels of results according to complexity. This paper applies the idea to a text classification task on 20 Newsgroups data set with two level of hierarchical targets and a comparison is made with training without the use of hidden layer branches.