FMiFood: Multi-modal Contrastive Learning for Food Image Classification
This addresses food image classification for dietary assessment, which is incremental as it builds on multi-modal contrastive learning with text enhancements.
The paper tackles the challenge of food image classification where intra-class diversity and inter-class similarity hinder performance, by introducing FMiFood, a multi-modal contrastive learning framework that integrates contextual text descriptions to enhance accuracy. The method shows improved performance on UPMC-101 and VFN datasets compared to existing methods.
Food image classification is the fundamental step in image-based dietary assessment, which aims to estimate participants' nutrient intake from eating occasion images. A common challenge of food images is the intra-class diversity and inter-class similarity, which can significantly hinder classification performance. To address this issue, we introduce a novel multi-modal contrastive learning framework called FMiFood, which learns more discriminative features by integrating additional contextual information, such as food category text descriptions, to enhance classification accuracy. Specifically, we propose a flexible matching technique that improves the similarity matching between text and image embeddings to focus on multiple key information. Furthermore, we incorporate the classification objectives into the framework and explore the use of GPT-4 to enrich the text descriptions and provide more detailed context. Our method demonstrates improved performance on both the UPMC-101 and VFN datasets compared to existing methods.