fugashi, a Tool for Tokenizing Japanese in Python
This addresses the exclusion of Japanese in multilingual NLP projects due to its complex tokenization requirements, though it is incremental as it builds on existing tools.
The paper tackles the problem of tokenizing Japanese text in NLP by introducing fugashi, a Python wrapper for MeCab, which simplifies usage and provides English documentation to make Japanese tokenization more accessible.
Recent years have seen an increase in the number of large-scale multilingual NLP projects. However, even in such projects, languages with special processing requirements are often excluded. One such language is Japanese. Japanese is written without spaces, tokenization is non-trivial, and while high quality open source tokenizers exist they can be hard to use and lack English documentation. This paper introduces fugashi, a MeCab wrapper for Python, and gives an introduction to tokenizing Japanese.