ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition
This work addresses a domain-specific problem for wildlife monitoring by improving behavior recognition in chimpanzees, though it is incremental as it builds on existing vision-language models.
The paper tackles chimpanzee behavior recognition from camera trap videos by enhancing a vision-language model with text descriptions of behaviors, achieving state-of-the-art performance with a +6.34% top-1 accuracy improvement on PanAf500 and gains in mean average precision on PanAf20K.
We show that chimpanzee behaviour understanding from camera traps can be enhanced by providing visual architectures with access to an embedding of text descriptions that detail species behaviours. In particular, we present a vision-language model which employs multi-modal decoding of visual features extracted directly from camera trap videos to process query tokens representing behaviours and output class predictions. Query tokens are initialised using a standardised ethogram of chimpanzee behaviour, rather than using random or name-based initialisations. In addition, the effect of initialising query tokens using a masked language model fine-tuned on a text corpus of known behavioural patterns is explored. We evaluate our system on the PanAf500 and PanAf20K datasets and demonstrate the performance benefits of our multi-modal decoding approach and query initialisation strategy on multi-class and multi-label recognition tasks, respectively. Results and ablations corroborate performance improvements. We achieve state-of-the-art performance over vision and vision-language models in top-1 accuracy (+6.34%) on PanAf500 and overall (+1.1%) and tail-class (+2.26%) mean average precision on PanAf20K. We share complete source code and network weights for full reproducibility of results and easy utilisation.