Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing
This work addresses tokenization optimization for downstream tasks, but it is incremental as it builds on existing methods with a focus on vocabulary restriction and broader applicability.
The paper tackles the problem of optimizing tokenization for already trained downstream models by generating tokenization results that lower loss values and training a tokenizer to reproduce these results, showing performance improvements in Japanese, Chinese, and English text classification tasks.
This paper proposes a method to optimize tokenization for the performance improvement of already trained downstream models. Our method generates tokenization results attaining lower loss values of a given downstream model on the training data for restricting vocabularies and trains a tokenizer reproducing the tokenization results. Therefore, our method can be applied to variety of tokenization methods, while existing work cannot due to the simultaneous learning of the tokenizer and the downstream model. This paper proposes an example of the BiLSTM-based tokenizer with vocabulary restriction, which can capture wider contextual information for the tokenization process than non-neural-based tokenization methods used in existing work. Experimental results on text classification in Japanese, Chinese, and English text classification tasks show that the proposed method improves performance compared to the existing methods for tokenization optimization.