From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding
This addresses the problem of vocabulary constraints in natural language understanding for AI applications, offering an incremental improvement over existing tokenization methods.
The paper tackles the limitation of fixed vocabularies in language models by introducing a hierarchical open-vocabulary model that operates on characters with word-level awareness, resulting in outperforming baselines on downstream tasks and demonstrating robustness to textual corruption and domain shift.
Current state-of-the-art models for natural language understanding require a preprocessing step to convert raw text into discrete tokens. This process known as tokenization relies on a pre-built vocabulary of words or sub-word morphemes. This fixed vocabulary limits the model's robustness to spelling errors and its capacity to adapt to new domains. In this work, we introduce a novel open-vocabulary language model that adopts a hierarchical two-level approach: one at the word level and another at the sequence level. Concretely, we design an intra-word module that uses a shallow Transformer architecture to learn word representations from their characters, and a deep inter-word Transformer module that contextualizes each word representation by attending to the entire word sequence. Our model thus directly operates on character sequences with explicit awareness of word boundaries, but without biased sub-word or word-level vocabulary. Experiments on various downstream tasks show that our method outperforms strong baselines. We also demonstrate that our hierarchical model is robust to textual corruption and domain shift.