MorphPiece : A Linguistic Tokenizer for Large Language Models
This addresses the issue of tokenization for NLP researchers and practitioners by offering a linguistically informed alternative to statistical tokenizers, though it appears incremental compared to existing morphological approaches.
The authors tackled the problem of tokenization in large language models by introducing MorphPiece, a linguistically motivated tokenizer based on morphological segmentation. They found that a GPT-style model trained with MorphPiece (MorphGPT) outperformed GPT-2 on various NLP tasks, often by a considerable margin, despite using about half the training iterations.
Tokenization is a critical part of modern NLP pipelines. However, contemporary tokenizers for Large Language Models are based on statistical analysis of text corpora, without much consideration to the linguistic features. I propose a linguistically motivated tokenization scheme, MorphPiece, which is based partly on morphological segmentation of the underlying text. A GPT-style causal language model trained on this tokenizer (called MorphGPT) shows comparable or superior performance on a variety of supervised and unsupervised NLP tasks, compared to the OpenAI GPT-2 model. Specifically I evaluated MorphGPT on language modeling tasks, zero-shot performance on GLUE Benchmark with various prompt templates, massive text embedding benchmark (MTEB) for supervised and unsupervised performance, and lastly with another morphological tokenization scheme (FLOTA, Hoffmann et al., 2022) and find that the model trained on MorphPiece outperforms GPT-2 on most evaluations, at times with considerable margin, despite being trained for about half the training iterations.