Coloring Deep CNN Layers with Activation Hue Loss
This work addresses the challenge of enhancing CNN regularization for more effective learning, though it appears incremental with modest gains.
The paper tackled the problem of improving deep CNN learning by modeling activation space structure with a novel activation hue parameter, and found that training with a combined one-hot and activation hue loss modestly improved classification performance across various tasks including ImageNet.
This paper proposes a novel hue-like angular parameter to model the structure of deep convolutional neural network (CNN) activation space, referred to as the {\em activation hue}, for the purpose of regularizing models for more effective learning. The activation hue generalizes the notion of color hue angle in standard 3-channel RGB intensity space to $N$-channel activation space. A series of observations based on nearest neighbor indexing of activation vectors with pre-trained networks indicate that class-informative activations are concentrated about an angle $θ$ in both the $(x,y)$ image plane and in multi-channel activation space. A regularization term in the form of hue-like angular $θ$ labels is proposed to complement standard one-hot loss. Training from scratch using combined one-hot + activation hue loss improves classification performance modestly for a wide variety of classification tasks, including ImageNet.