Tackling Early Sparse Gradients in Softmax Activation Using Leaky Squared Euclidean Distance
This work tackles a specific issue in one-shot learning for computer vision, offering an incremental improvement to gradient handling in softmax-based models.
The paper addresses the early sparse gradients problem in softmax activation when using squared Euclidean distance in one-shot learning, which can hinder model improvement. The proposed leaky squared Euclidean distance method improves one-shot classification accuracy on Omniglot and miniImageNet datasets.
Softmax activation is commonly used to output the probability distribution over categories based on certain distance metric. In scenarios like one-shot learning, the distance metric is often chosen to be squared Euclidean distance between the query sample and the category prototype. This practice works well in most time. However, we find that choosing squared Euclidean distance may cause distance explosion leading gradients to be extremely sparse in the early stage of back propagation. We term this phenomena as the early sparse gradients problem. Though it doesn't deteriorate the convergence of the model, it may set up a barrier to further model improvement. To tackle this problem, we propose to use leaky squared Euclidean distance to impose a restriction on distances. In this way, we can avoid distance explosion and increase the magnitude of gradients. Extensive experiments are conducted on Omniglot and miniImageNet datasets. We show that using leaky squared Euclidean distance can improve one-shot classification accuracy on both datasets.