DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering
This addresses the problem of limited culturally grounded data for Japanese multimodal AI, enabling better model performance in that domain, though it is incremental as it builds on existing dataset creation methods.
The authors tackled the scarcity of high-quality Japanese vision-and-language resources by creating DEJIMA, a dataset with 3.88M image-text pairs for captioning and VQA, which outperforms existing datasets in Japaneseness and naturalness while maintaining factual correctness and improving model performance on benchmarks.
This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.