Generating Diverse and Informative Natural Language Fashion Feedback
This work addresses the need for more informative fashion feedback in applications like smart devices, though it is incremental as it builds on existing image captioning methods.
The paper tackles the problem of generating natural language fashion feedback on outfit images, resulting in more diverse and detailed responses by using a Maximum Mutual Information decoding technique, with evaluation showing improvements in language metrics and human assessments.
Recent advances in multi-modal vision and language tasks enable a new set of applications. In this paper, we consider the task of generating natural language fashion feedback on outfit images. We collect a unique dataset, which contains outfit images and corresponding positive and constructive fashion feedback. We treat each feedback type separately, and train deep generative encoder-decoder models with visual attention, similar to the standard image captioning pipeline. Following this approach, the generated sentences tend to be too general and non-informative. We propose an alternative decoding technique based on the Maximum Mutual Information objective function, which leads to more diverse and detailed responses. We evaluate our model with common language metrics, and also show human evaluation results. This technology is applied within the ``Alexa, how do I look?'' feature, publicly available in Echo Look devices.