Specialised or Generic? Tokenization Choices for Radiology Language Models
This work addresses the under-explored impact of tokenization in radiology, offering practical benefits for making language models more accessible and effective in healthcare settings, though it is incremental as it systematically compares existing tokenizer types.
The study tackled the problem of tokenizer choice for radiology language models by comparing general, medical, and domain-specific tokenizers on radiology report summarization across three imaging modalities, finding that domain-specific tokenizers outperformed others when trained from scratch and reduced memory requirements due to smaller vocabularies and shorter sequences.
The vocabulary used by language models (LM) - defined by the tokenizer - plays a key role in text generation quality. However, its impact remains under-explored in radiology. In this work, we address this gap by systematically comparing general, medical, and domain-specific tokenizers on the task of radiology report summarisation across three imaging modalities. We also investigate scenarios with and without LM pre-training on PubMed abstracts. Our findings demonstrate that medical and domain-specific vocabularies outperformed widely used natural language alternatives when models are trained from scratch. Pre-training partially mitigates performance differences between tokenizers, whilst the domain-specific tokenizers achieve the most favourable results. Domain-specific tokenizers also reduce memory requirements due to smaller vocabularies and shorter sequences. These results demonstrate that adapting the vocabulary of LMs to the clinical domain provides practical benefits, including improved performance and reduced computational demands, making such models more accessible and effective for both research and real-world healthcare settings.