Sign Language Recognition in the Age of LLMs
This work provides an initial assessment of zero-shot ISLR using VLMs, highlighting current limitations and the potential of scaling for sign language recognition.
The paper investigates whether general-purpose Vision Language Models (VLMs) can perform isolated sign language recognition (ISLR) without task-specific training. Results show that open-source VLMs lag behind supervised classifiers, but larger proprietary models achieve higher accuracy, indicating the importance of model scale and data diversity.
Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.