CLIP-HandID: Vision-Language Model for Hand-Based Person Identification
It addresses a critical problem in criminal investigations, such as sexual abuse, where hand images are often the only evidence, though it is incremental as it adapts existing models.
This paper tackles person identification from hand images for criminal investigations by proposing CLIP-HandID, which uses a pre-trained vision-language model with textual prompts and pseudo-tokens, achieving significant performance improvements over existing methods on multi-ethnic datasets.
This paper introduces a novel approach to person identification using hand images, designed specifically for criminal investigations. The method is particularly valuable in serious crimes such as sexual abuse, where hand images are often the only identifiable evidence available. Our proposed method, CLIP-HandID, leverages a pre-trained foundational vision-language model - CLIP - to efficiently learn discriminative deep feature representations from hand images (input to CLIP's image encoder) using textual prompts as semantic guidance. Since hand images are labeled with indexes rather than text descriptions, we employ a textual inversion network to learn pseudo-tokens that encode specific visual contexts or appearance attributes. These learned pseudo-tokens are then incorporated into textual prompts, which are fed into CLIP's text encoder to leverage its multi-modal reasoning and enhance generalization for identification. Through extensive evaluations on two large, publicly available hand datasets with multi-ethnic representation, we demonstrate that our method significantly outperforms existing approaches.