UMDFood: Vision-language models boost food composition compilation
This addresses the need for timely and accurate nutrition data in precision nutrition and the food industry, representing a novel application of vision-language models to a specific domain.
The authors tackled the problem of laborious and delayed food composition compilation by proposing UMDFood-VL, a vision-language model that uses front-of-package labeling and product images, achieving a macro-AUCROC of 0.921 for fat content estimation and up to 82.2% of products with estimation errors under 10% compared to chemical analysis.
Nutrition information is crucial in precision nutrition and the food industry. The current food composition compilation paradigm relies on laborious and experience-dependent methods. However, these methods struggle to keep up with the dynamic consumer market, resulting in delayed and incomplete nutrition data. In addition, earlier machine learning methods overlook the information in food ingredient statements or ignore the features of food images. To this end, we propose a novel vision-language model, UMDFood-VL, using front-of-package labeling and product images to accurately estimate food composition profiles. In order to empower model training, we established UMDFood-90k, the most comprehensive multimodal food database to date, containing 89,533 samples, each labeled with image and text-based ingredient descriptions and 11 nutrient annotations. UMDFood-VL achieves the macro-AUCROC up to 0.921 for fat content estimation, which is significantly higher than existing baseline methods and satisfies the practical requirements of food composition compilation. Meanwhile, up to 82.2% of selected products' estimated error between chemical analysis results and model estimation results are less than 10%. This performance sheds light on generalization towards other food and nutrition-related data compilation and catalyzation for the evolution of generative AI-based technology in other food applications that require personalization.