When a Zero-Shooter Cheats: Improving Age Estimation via Activation Steering
For researchers and practitioners using VLMs for age estimation, this work addresses a critical failure mode that inflates benchmark performance but fails in real-world scenarios.
The paper identifies that vision-language models (VLMs) use an identity shortcut for zero-shot age estimation, inferring age from memorized identity rather than visual features, leading to errors for non-celebrities. They propose an activation steering method that reduces mean absolute error by up to 25% across benchmarks.
Different age-related regulations have been proposed to protect minors from harmful content and interactions online. Automated age estimation is central to enforcing such regulations, and vision-language models (VLMs) achieve state-of-the-art performance on this task. However, we find that the zero-shot nature of VLM-based age estimation produces an unexpected side effect we call the identity shortcut: Instead of estimating age from visual features, VLMs tend to identify the depicted person and infer their age from memorized knowledge. This phenomenon leads to substantially incorrect predictions when non-celebrities are misidentified as celebrities. It also produces deceptively high robustness to noise and adversarial perturbations on celebrity images, which dominate popular benchmarks. To mitigate this, we propose an activation steering method that suppresses the shortcut by intervening on the hidden states of the VLM. This method improves age estimation accuracy for both memorized and unseen identities, reducing mean absolute error by up to 25% across popular benchmarks.