Multiscale sequence modeling with a learned dictionary
This work addresses language modeling efficiency for smaller models, offering a hybrid approach between character and word levels, but it is incremental as it builds on existing methods like BPE and LSTMs.
The authors tackled sequence modeling by proposing a multi-scale model that predicts overlapping multi-symbol tokens using a learned dictionary based on byte-pair encoding, achieving better performance than a regular LSTM on language modeling tasks, particularly for smaller models.
We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that the model is trained with. When applied to language modelling, our model has the flexibility of character-level models while maintaining many of the performance benefits of word-level models. Our experiments show that this model performs better than a regular LSTM on language modeling tasks, especially for smaller models.