Exploiting the Experts: Unauthorized Compression in MoE-LLMs
This work addresses a security problem for developers and users of MoE-LLMs, highlighting a novel threat vector and providing a framework for secure specialization, though it is incremental in building on existing MoE architectures.
The paper tackles the vulnerability of Mixture-of-Experts (MoE) large language models to unauthorized compression by adversaries, finding that pruning experts leads to significant degradation in task accuracy without targeted re-alignment, and proposes defense strategies to resist such attacks.
Mixture-of-Experts (MoE) architectures are increasingly adopted in large language models (LLMs) for their scalability and efficiency. However, their modular structure introduces a unique vulnerability: adversaries can attempt to compress or repurpose models by pruning experts and cheaply fine-tuning the remainder, effectively bypassing licensing and security constraints. In this paper, we systematically study the prunability of MoE-LLMs under task-specific usage. We first develop an expert attribution framework that identifies the subset of experts most responsible for a given task, then evaluate the performance trade-offs of pruning and re-aligning these experts using active learning-driven fine-tuning. Our findings reveal a critical knowledge loss--recovery trade-off: while certain experts can be isolated to retain task accuracy, significant degradation occurs without targeted re-alignment. Based on this analysis, we propose defense strategies that aim to make MoE models harder to compress and fine-tune without authorization, including entangled expert training and selective fine-tuning protocols that resist unauthorized adaptation. By positioning expert pruning as both a threat vector and a defense target, this work highlights the dual-use nature of MoE modularity and provides the first systematic evaluation framework for secure specialization of MoE-LLMs.