What should I wear to a party in a Greek taverna? Evaluation for Conversational Agents in the Fashion Domain
This addresses the need for better evaluation methods for conversational agents in online fashion retail, though it is incremental as it focuses on dataset creation rather than a new method.
The authors tackled the problem of evaluating LLM-powered conversational agents in fashion e-commerce by creating a multilingual dataset of 4k conversations to measure their ability to interface with backend systems, showing it scales to business needs and aids iterative tool development.
Large language models (LLMs) are poised to revolutionize the domain of online fashion retail, enhancing customer experience and discovery of fashion online. LLM-powered conversational agents introduce a new way of discovery by directly interacting with customers, enabling them to express in their own ways, refine their needs, obtain fashion and shopping advice that is relevant to their taste and intent. For many tasks in e-commerce, such as finding a specific product, conversational agents need to convert their interactions with a customer to a specific call to different backend systems, e.g., a search system to showcase a relevant set of products. Therefore, evaluating the capabilities of LLMs to perform those tasks related to calling other services is vital. However, those evaluations are generally complex, due to the lack of relevant and high quality datasets, and do not align seamlessly with business needs, amongst others. To this end, we created a multilingual evaluation dataset of 4k conversations between customers and a fashion assistant in a large e-commerce fashion platform to measure the capabilities of LLMs to serve as an assistant between customers and a backend engine. We evaluate a range of models, showcasing how our dataset scales to business needs and facilitates iterative development of tools.