Generation with Dynamic Vocabulary
This addresses efficiency and quality bottlenecks in text generation for various downstream applications, though it appears incremental as an enhancement to existing token-based approaches.
The paper tackles the problem of limited vocabulary flexibility in language models by introducing a dynamic vocabulary that uses arbitrary text spans as generation units, resulting in a 25% improvement in MAUVE metric and 20% reduction in latency compared to standard models.
We introduce a new dynamic vocabulary for language models. It can involve arbitrary text spans during generation. These text spans act as basic generation bricks, akin to tokens in the traditional static vocabularies. We show that, the ability to generate multi-tokens atomically improve both generation quality and efficiency (compared to the standard language model, the MAUVE metric is increased by 25%, the latency is decreased by 20%). The dynamic vocabulary can be deployed in a plug-and-play way, thus is attractive for various downstream applications. For example, we demonstrate that dynamic vocabulary can be applied to different domains in a training-free manner. It also helps to generate reliable citations in question answering tasks (substantially enhancing citation results without compromising answer accuracy).