Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI
This work addresses the problem of scalable and controllable AI for firms in marketing, offering an incremental improvement over existing distillation methods.
The paper tackles the challenge of deploying large language models in interactive, multi-turn customer service settings by shifting knowledge distillation from output imitation to contextual guidance, resulting in improved service quality and customer satisfaction while maintaining policy alignment.
As large language models increasingly mediate firm - customer interactions, firms face a tradeoff: the most capable models perform well but are costly and difficult to control at scale. Existing knowledge distillation methods address this challenge by training weaker, deployable models to imitate frontier outputs; however, such open-loop approaches are poorly suited to interactive, multi-turn settings where responses must be sequenced coherently across conversational states. We propose a shift in what knowledge is distilled - from output imitation to contextual guidance. We develop a framework in which a superior teacher model constructs a reusable library of strategic textual guidance for particular scenarios likely to be encountered by the student. When deployed, the student retrieves the context-specific guidance at inference time, enabling adaptive behavior without retraining. Using customer-service interactions, we show that this approach improves service quality and customer satisfaction relative to standard fine-tuning while maintaining alignment with firm policies. The results position inference-time textual guidance as a scalable and controllable approach to distillation for interactive AI agents in marketing settings.