UniCase -- Rethinking Casing in Language Models
This addresses the issue of inconsistent casing in text data for natural language processing applications, but it is incremental as it builds on existing models like RoBERTa.
The authors tackled the problem of case-sensitivity in language models by proposing UniCase, a modified RoBERTa architecture with a new tokenization strategy, which improved performance on the GLUE benchmark by 0.42 points and showed a 5.88-point gain on fully uppercased text.
In this paper, we introduce a new approach to dealing with the problem of case-sensitiveness in Language Modelling (LM). We propose simple architecture modification to the RoBERTa language model, accompanied by a new tokenization strategy, which we named Unified Case LM (UniCase). We tested our solution on the GLUE benchmark, which led to increased performance by 0.42 points. Moreover, we prove that the UniCase model works much better when we have to deal with text data, where all tokens are uppercased (+5.88 point).