Leveraging Large Language Models for Multimodal Search
This addresses the problem of improving multimodal search for users in domains like shopping, though it appears incremental as it builds on existing datasets and interfaces.
The paper tackles the problem of unreliable multimodal search systems by introducing a novel model that achieves a new performance milestone on the Fashion200K dataset, and proposes an LLM-based interface for natural language interaction to enhance the search experience.
Multimodal search has become increasingly important in providing users with a natural and effective way to ex-press their search intentions. Images offer fine-grained details of the desired products, while text allows for easily incorporating search modifications. However, some existing multimodal search systems are unreliable and fail to address simple queries. The problem becomes harder with the large variability of natural language text queries, which may contain ambiguous, implicit, and irrelevant in-formation. Addressing these issues may require systems with enhanced matching capabilities, reasoning abilities, and context-aware query parsing and rewriting. This paper introduces a novel multimodal search model that achieves a new performance milestone on the Fashion200K dataset. Additionally, we propose a novel search interface integrating Large Language Models (LLMs) to facilitate natural language interaction. This interface routes queries to search systems while conversationally engaging with users and considering previous searches. When coupled with our multimodal search model, it heralds a new era of shopping assistants capable of offering human-like interaction and enhancing the overall search experience.