Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
This addresses a specific gap in document understanding for Japanese language users, as vertical writing is common in Japanese documents, but the work is incremental as it focuses on evaluating and improving existing models rather than introducing new methods.
The study evaluated multimodal large language models (MLLMs) on vertically written Japanese text, finding they performed worse on vertical text than horizontal text, and showed that fine-tuning on a synthetic dataset improved performance for models that previously could not handle vertical writing.
Multimodal Large Language Models (MLLMs) have seen rapid advances in recent years and are now being applied to visual document understanding tasks. They are expected to process a wide range of document images across languages, including Japanese. Understanding documents from images requires models to read what are written in them. Since some Japanese documents are written vertically, support for vertical writing is essential. However, research specifically focused on vertically written Japanese text remains limited. In this study, we evaluate the reading capability of existing MLLMs on vertically written Japanese text. First, we generate a synthetic Japanese OCR dataset by rendering Japanese texts into images, and use it for both model fine-tuning and evaluation. This dataset includes Japanese text in both horizontal and vertical writing. We also create an evaluation dataset sourced from the real-world document images containing vertically written Japanese text. Using these datasets, we demonstrate that the existing MLLMs perform worse on vertically written Japanese text than on horizontally written Japanese text. Furthermore, we show that training MLLMs on our synthesized Japanese OCR dataset results in improving the performance of models that previously could not handle vertical writing. The datasets and code are publicly available https://github.com/llm-jp/eval_vertical_ja.