Cyclical Focal Loss
This work addresses the challenge of class imbalance in machine learning, offering a more universal loss function that improves performance without increasing training time, though it appears incremental as an enhancement to existing focal loss methods.
The authors tackled the problem of training deep neural networks on datasets with varying class balance by introducing a cyclical focal loss, which outperformed cross-entropy softmax and focal loss across balanced, imbalanced, and long-tailed datasets in experiments on CIFAR-10/100, ImageNet, and Open Long-Tailed Recognition benchmarks.
The cross-entropy softmax loss is the primary loss function used to train deep neural networks. On the other hand, the focal loss function has been demonstrated to provide improved performance when there is an imbalance in the number of training samples in each class, such as in long-tailed datasets. In this paper, we introduce a novel cyclical focal loss and demonstrate that it is a more universal loss function than cross-entropy softmax loss or focal loss. We describe the intuition behind the cyclical focal loss and our experiments provide evidence that cyclical focal loss provides superior performance for balanced, imbalanced, or long-tailed datasets. We provide numerous experimental results for CIFAR-10/CIFAR-100, ImageNet, balanced and imbalanced 4,000 training sample versions of CIFAR-10/CIFAR-100, and ImageNet-LT and Places-LT from the Open Long-Tailed Recognition (OLTR) challenge. Implementing the cyclical focal loss function requires only a few lines of code and does not increase training time. In the spirit of reproducibility, our code is available at \url{https://github.com/lnsmith54/CFL}.