ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation
This addresses the problem of fragmented single-task models in food computing for applications like culinary AI, though it is incremental as it builds on existing LLMs and image models.
The paper tackles the lack of integrated multimodal approaches in food computing by introducing ChefFusion, a foundation model that simultaneously handles tasks like recipe and food image generation, achieving superior performance in these areas.
Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub.