Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries
This work addresses the challenge of high intra-class and low inter-class variability in meal images for nutrition estimation, though it is incremental as it builds on existing attention-based methods for a specific domain.
The paper tackles the problem of recipe retrieval from images to estimate nutrient content, proposing a model that uses self-attention and ingredient-attention mechanisms to process raw recipe text and improve retrieval by focusing on specific preparation steps, achieving desirable results compared to two baseline methods.
Direct computer vision based-nutrient content estimation is a demanding task, due to deformation and occlusions of ingredients, as well as high intra-class and low inter-class variability between meal classes. In order to tackle these issues, we propose a system for recipe retrieval from images. The recipe information can subsequently be used to estimate the nutrient content of the meal. In this study, we utilize the multi-modal Recipe1M dataset, which contains over 1 million recipes accompanied by over 13 million images. The proposed model can operate as a first step in an automatic pipeline for the estimation of nutrition content by supporting hints related to ingredient and instruction. Through self-attention, our model can directly process raw recipe text, making the upstream instruction sentence embedding process redundant and thus reducing training time, while providing desirable retrieval results. Furthermore, we propose the use of an ingredient attention mechanism, in order to gain insight into which instructions, parts of instructions or single instruction words are of importance for processing a single ingredient within a certain recipe. Attention-based recipe text encoding contributes to solving the issue of high intra-class/low inter-class variability by focusing on preparation steps specific to the meal. The experimental results demonstrate the potential of such a system for recipe retrieval from images. A comparison with respect to two baseline methods is also presented.