Tokenization Preference for Human and Machine Learning Model: An Annotation Study
This addresses tokenization alignment issues for NLP researchers and practitioners, but it is incremental as it builds on existing tokenization methods.
The study investigated whether tokenization preferences for humans align with those for machine learning models, using a Japanese commonsense question-answering dataset with six tokenizers, and found that they are not always the same, with language model-based methods potentially serving as a compromise.
Is preferred tokenization for humans also preferred for machine-learning (ML) models? This study examines the relations between preferred tokenization for humans (appropriateness and readability) and one for ML models (performance on an NLP task). The question texts of the Japanese commonsense question-answering dataset are tokenized with six different tokenizers, and the performances of human annotators and ML models were compared. Furthermore, we analyze relations among performance of answers by human and ML model, the appropriateness of tokenization for human, and response time to questions by human. This study provides a quantitative investigation result that shows that preferred tokenizations for humans and ML models are not necessarily always the same. The result also implies that existing methods using language models for tokenization could be a good compromise both for human and ML models.