EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata
This work addresses image forensics and calibration challenges for applications in security and media analysis, though it is incremental as it builds on existing multimodal learning techniques.
The paper tackles the problem of learning visual representations that capture camera-specific information by training a multimodal embedding between image patches and EXIF metadata, achieving significant performance improvements on image forensics and calibration tasks, such as zero-shot localization of spliced regions.
We learn a visual representation that captures information about the camera that recorded a given photo. To do this, we train a multimodal embedding between image patches and the EXIF metadata that cameras automatically insert into image files. Our model represents this metadata by simply converting it to text and then processing it with a transformer. The features that we learn significantly outperform other self-supervised and supervised features on downstream image forensics and calibration tasks. In particular, we successfully localize spliced image regions "zero shot" by clustering the visual embeddings for all of the patches within an image.