FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
This work addresses the practical need for accurate dietary management tools in Japanese food culture, but it is incremental as it applies existing MLLM methods to a specific language and dataset.
The paper tackled the problem of generating Japanese recipes from food images by fine-tuning open multimodal large language models (LLaVA-1.5 and Phi-3 Vision) on a Japanese recipe dataset, achieving an F1 score of 0.531 in ingredient generation, which outperformed GPT-4o's 0.481, while showing comparable performance in cooking procedure generation.
Research on food image understanding using recipe data has been a long-standing focus due to the diversity and complexity of the data. Moreover, food is inextricably linked to people's lives, making it a vital research area for practical applications such as dietary management. Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities, not only in their vast knowledge but also in their ability to handle languages naturally. While English is predominantly used, they can also support multiple languages including Japanese. This suggests that MLLMs are expected to significantly improve performance in food image understanding tasks. We fine-tuned open MLLMs LLaVA-1.5 and Phi-3 Vision on a Japanese recipe dataset and benchmarked their performance against the closed model GPT-4o. We then evaluated the content of generated recipes, including ingredients and cooking procedures, using 5,000 evaluation samples that comprehensively cover Japanese food culture. Our evaluation demonstrates that the open models trained on recipe data outperform GPT-4o, the current state-of-the-art model, in ingredient generation. Our model achieved F1 score of 0.531, surpassing GPT-4o's F1 score of 0.481, indicating a higher level of accuracy. Furthermore, our model exhibited comparable performance to GPT-4o in generating cooking procedure text.