SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models
This addresses the issue of negative transfer in social perception tasks for AI systems, which is incremental as it builds on existing VLM limitations.
The paper tackled the problem of 'social degradation' in pre-trained vision-language models, where general pre-training impairs the visual encoder's ability to represent nuanced social information, and proposed SocialFusion, a unified framework that achieved positive transfer across five social tasks and comparable performance to task-specific state-of-the-art models.
Understanding social interactions from visual cues is a fundamental challenge for a socially competent AI. While powerful pre-trained vision-language models (VLMs) have shown remarkable general capabilities, they surprisingly struggle to unify and learn multiple social perception tasks simultaneously, often exhibiting negative transfer. We identify that this negative transfer stems from a critical issue we term "social degradation," whereby the general visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information. We investigate this behavior further under two lenses: decodability through linear representation probing and compatibility through gradient conflict analysis, revealing that both play a role in the degradation, especially the former, which is significantly compromised in the VLM pre-training process. To address these issues, we propose SocialFusion, a unified framework that learns a minimal connection between a frozen visual encoder and a language model. Compared with existing VLMs, it exhibits positive transfer across all five social tasks, leveraging synergies between them to enhance overall performance and achieves comparable performance to task-specific state-of-the-art models on various benchmarks. Our findings suggest that current VLM pre-training strategies may be detrimental to acquiring general social competence and highlight the need for more socially-aware training paradigms.