Partially Does It: Towards Scene-Level FG-SBIR with Partial Input
This addresses a critical bottleneck in scene-level sketch research for applications like image retrieval, though it is incremental as it builds on existing methods to handle partial inputs.
The paper tackles the problem of scene-level fine-grained sketch-based image retrieval (FG-SBIR) when sketches are partial, noting that existing methods fail as sketches become more incomplete. It proposes a set-based approach using optimal transport to model cross-modal region associativity, achieving state-of-the-art performance on existing datasets.
We scrutinise an important observation plaguing scene-level sketch research -- that a significant portion of scene sketches are "partial". A quick pilot study reveals: (i) a scene sketch does not necessarily contain all objects in the corresponding photo, due to the subjective holistic interpretation of scenes, (ii) there exists significant empty (white) regions as a result of object-level abstraction, and as a result, (iii) existing scene-level fine-grained sketch-based image retrieval methods collapse as scene sketches become more partial. To solve this "partial" problem, we advocate for a simple set-based approach using optimal transport (OT) to model cross-modal region associativity in a partially-aware fashion. Importantly, we improve upon OT to further account for holistic partialness by comparing intra-modal adjacency matrices. Our proposed method is not only robust to partial scene-sketches but also yields state-of-the-art performance on existing datasets.