Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference
This addresses generalization issues in natural language inference for researchers, but it is incremental as it builds on existing focal loss methods.
The paper tackled the problem of deep neural networks using shallow heuristics in datasets, leading to poor generalization, by exploring focal loss as a modulated cross-entropy to constrain heuristic use. The result showed focal loss improved out-of-distribution accuracy but slightly decreased in-distribution performance, while being inferior to methods like unbiased focal loss and self-debiasing ensembles.
There is no such thing as a perfect dataset. In some datasets, deep neural networks discover underlying heuristics that allow them to take shortcuts in the learning process, resulting in poor generalization capability. Instead of using standard cross-entropy, we explore whether a modulated version of cross-entropy called focal loss can constrain the model so as not to use heuristics and improve generalization performance. Our experiments in natural language inference show that focal loss has a regularizing impact on the learning process, increasing accuracy on out-of-distribution data, but slightly decreasing performance on in-distribution data. Despite the improved out-of-distribution performance, we demonstrate the shortcomings of focal loss and its inferiority in comparison to the performance of methods such as unbiased focal loss and self-debiasing ensembles.