Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality
This addresses robustness issues in language models for social media text, though it is an incremental improvement over existing subword methods.
The paper tackled the problem of subword tokenization's inability to handle infrequent spellings and character-level alterations by proposing a character-based subword module (char2subword) that learns embeddings from characters, resulting in significant performance improvements on the LinCE benchmark for social media linguistic code-switching.
Byte-pair encoding (BPE) is a ubiquitous algorithm in the subword tokenization process of language models as it provides multiple benefits. However, this process is solely based on pre-training data statistics, making it hard for the tokenizer to handle infrequent spellings. On the other hand, though robust to misspellings, pure character-level models often lead to unreasonably long sequences and make it harder for the model to learn meaningful words. To alleviate these challenges, we propose a character-based subword module (char2subword) that learns the subword embedding table in pre-trained models like BERT. Our char2subword module builds representations from characters out of the subword vocabulary, and it can be used as a drop-in replacement of the subword embedding table. The module is robust to character-level alterations such as misspellings, word inflection, casing, and punctuation. We integrate it further with BERT through pre-training while keeping BERT transformer parameters fixed--and thus, providing a practical method. Finally, we show that incorporating our module to mBERT significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark.