Can Modern Vision Models Understand the Difference Between an Object and a Look-alike?
This addresses a subtle gap in computer vision for AI systems needing fine-grained perceptual understanding, though it is incremental as it builds on existing CLIP capabilities.
The study investigated whether vision-language models like CLIP can distinguish between real objects and lookalikes, finding that estimating a direction in CLIP's embedding space improved discrimination in cross-modal retrieval and enhanced caption quality.
Recent advances in computer vision have yielded models with strong performance on recognition benchmarks; however, significant gaps remain in comparison to human perception. One subtle ability is to judge whether an image looks like a given object without being an instance of that object. We study whether vision-language models such as CLIP capture this distinction. We curated a dataset named RoLA (Real or Lookalike) of real and lookalike exemplars (e.g., toys, statues, drawings, pareidolia) across multiple categories, and first evaluate a prompt-based baseline with paired "real"/"lookalike" prompts. We then estimate a direction in CLIP's embedding space that moves representations between real and lookalike. Applying this direction to image and text embeddings improves discrimination in cross-modal retrieval on Conceptual12M, and also enhances captions produced by a CLIP prefix captioner.