Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective
This work addresses the intellectual property protection of proprietary LLMs against distillation attacks, focusing on a previously underexplored logit-based approach, which is incremental but important for model security.
The paper tackles the problem of logit-based distillation attacks on proprietary large language models (LLMs) by proposing an information-theoretic defense that minimizes conditional mutual information to remove distillation-relevant information. The method significantly degrades distillation performance while preserving task accuracy, as demonstrated in experiments across multiple LLMs and distillation algorithms.
Proprietary large language models (LLMs) embody substantial economic value and are generally exposed only as black-box APIs, yet adversaries can still exploit their outputs to extract knowledge via distillation. Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored. In this work, we analyze this problem and present an effective solution from an information-theoretic perspective. We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels. This quantity captures contextual information beneficial for model extraction, motivating us to defend distillation via CMI minimization. Guided by our theoretical analysis, we propose learning a transformation matrix that purifies the original outputs to enhance distillation resistance. We further derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility. Extensive experiments across multiple LLMs and strong distillation algorithms demonstrate that the proposed method significantly degrades distillation performance while preserving task accuracy, effectively protecting models' intellectual property.