ELM: Embedding and Logit Margins for Long-Tail Learning
This addresses the challenge of poor generalization for tail classes in neural models, offering an incremental improvement by unifying existing margin and embedding regularization techniques.
The paper tackles the problem of long-tail learning under skewed label distributions by proposing ELM, a method that enforces margins in logit space and regularizes embeddings, which reduces the generalization gap and results in tighter tail class embeddings.
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners. Several recent approaches for the problem have proposed enforcing a suitable margin in logit space. Such techniques are intuitive analogues of the guiding principle behind SVMs, and are equally applicable to linear models and neural models. However, when applied to neural models, such techniques do not explicitly control the geometry of the learned embeddings. This can be potentially sub-optimal, since embeddings for tail classes may be diffuse, resulting in poor generalization for these classes. We present Embedding and Logit Margins (ELM), a unified approach to enforce margins in logit space, and regularize the distribution of embeddings. This connects losses for long-tail learning to proposals in the literature on metric embedding, and contrastive learning. We theoretically show that minimising the proposed ELM objective helps reduce the generalisation gap. The ELM method is shown to perform well empirically, and results in tighter tail class embeddings.