Group Preference Collapse in Personalized Multimodal Large Language Models
For developers of personalized AI systems, this work addresses a critical failure mode where multi-user personalization degrades to population-level responses, offering a practical solution to preserve individual preferences.
The paper identifies 'group preference collapse' in personalized multimodal LLMs, where models drift toward dominant population preferences and suppress individual user preferences. The proposed PrefMoE framework mitigates this by separating stable profile information from preference representations, achieving improved preference-sensitive personalization across multiple MLLM backbones.
Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse. Project page: https://prefmoe.github.io/.