CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval
This work addresses the problem of understanding the components of cooking recipes for food image-recipe association and retrieval tasks, which is an incremental improvement in the food domain.
This paper tackles the problem of understanding individual entities and their roles in multi-modal food data, specifically image-text pairs of cooking recipes. They introduce a cross-modal learning framework that uses tree-structured LSTMs to jointly model image and text representations, enabling the identification of main ingredients and cooking actions without explicit supervision, and learning more meaningful feature representations for cross-modal retrieval.
Despite the abundance of multi-modal data, such as image-text pairs, there has been little effort in understanding the individual entities and their different roles in the construction of these data instances. In this work, we endeavour to discover the entities and their corresponding importance in cooking recipes automaticall} as a visual-linguistic association problem. More specifically, we introduce a novel cross-modal learning framework to jointly model the latent representations of images and text in the food image-recipe association and retrieval tasks. This model allows one to discover complex functional and hierarchical relationships between images and text, and among textual parts of a recipe including title, ingredients and cooking instructions. Our experiments show that by making use of efficient tree-structured Long Short-Term Memory as the text encoder in our computational cross-modal retrieval framework, we are not only able to identify the main ingredients and cooking actions in the recipe descriptions without explicit supervision, but we can also learn more meaningful feature representations of food recipes, appropriate for challenging cross-modal retrieval and recipe adaption tasks.