Generative Knowledge Transfer for Neural Language Models
This addresses privacy concerns in language model adaptation for users, though it is incremental as it builds on existing knowledge transfer methods.
The paper tackles the problem of training neural language models without direct access to user data by proposing a generative knowledge transfer technique, where a student RNN learns from text and output probabilities generated by a teacher RNN, achieving privacy-conscious adaptation.
In this paper, we propose a generative knowledge transfer technique that trains an RNN based language model (student network) using text and output probabilities generated from a previously trained RNN (teacher network). The text generation can be conducted by either the teacher or the student network. We can also improve the performance by taking the ensemble of soft labels obtained from multiple teacher networks. This method can be used for privacy conscious language model adaptation because no user data is directly used for training. Especially, when the soft labels of multiple devices are aggregated via a trusted third party, we can expect very strong privacy protection.