Visual Product Graph: Bridging Visual Products And Composite Images For End-to-End Style Recommendations
This addresses the need for style recommendations in visual search systems, such as at Pinterest, by enabling navigation from products to composite scenes and complementary product suggestions, though it appears incremental with improvements to existing computer vision models.
The paper tackles the problem of retrieving visually distinct but semantically similar content in visual search by introducing Visual Product Graph (VPG), an online real-time system that achieves 78.8% extremely similar@1 in human relevance evaluations and a 6% module engagement rate.
Retrieving semantically similar but visually distinct contents has been a critical capability in visual search systems. In this work, we aim to tackle this problem with Visual Product Graph (VPG), leveraging high-performance infrastructure for storage and state-of-the-art computer vision models for image understanding. VPG is built to be an online real-time retrieval system that enables navigation from individual products to composite scenes containing those products, along with complementary recommendations. Our system not only offers contextual insights by showcasing how products can be styled in a context, but also provides recommendations for complementary products drawn from these inspirations. We discuss the essential components for building the Visual Product Graph, along with the core computer vision model improvements across object detection, foundational visual embeddings, and other visual signals. Our system achieves a 78.8% extremely similar@1 in end-to-end human relevance evaluations, and a 6% module engagement rate. The "Ways to Style It" module, powered by the Visual Product Graph technology, is deployed in production at Pinterest.