Bridging the Gap Between Object Detection and User Intent via Query-Modulation
This addresses the issue of misaligned detections for users interacting with objects via cameras, especially on mobile devices, though it is incremental as it builds on existing detection methods.
The paper tackles the problem of object detection models failing to align with user intent by introducing query-modulated detectors that incorporate user queries as embeddings, resulting in superior performance for query-based detection and even outperforming standard detectors on the COCO task.
When interacting with objects through cameras, or pictures, users often have a specific intent. For example, they may want to perform a visual search. With most object detection models relying on image pixels as their sole input, undesired results are not uncommon. Most typically: lack of a high-confidence detection on the object of interest, or detection with a wrong class label. The issue is especially severe when operating capacity-constrained mobile object detectors on-device. In this paper we investigate techniques to modulate mobile detectors to explicitly account for the user intent, expressed as an embedding of a simple query. Compared to standard detectors, query-modulated detectors show superior performance at detecting objects for a given user query. Thanks to large-scale training data synthesized from standard object detection annotations, query-modulated detectors also outperform a specialized referring expression recognition system. Query-modulated detectors can also be trained to simultaneously solve for both localizing a user query and standard detection, even outperforming standard mobile detectors at the canonical COCO task.