Neural Composition: Learning to Generate from Multiple Models
This addresses the need for more flexible and scalable personalization in language modeling, though it appears incremental as it builds on existing decomposition concepts.
The paper tackles the problem of adapting language models to context and personal preferences without requiring class-annotated data, by proposing a system that learns to activate and combine multiple model components from unlabeled text, achieving results that enable scalable adaptation.
Decomposing models into multiple components is critically important in many applications such as language modeling (LM) as it enables adapting individual components separately and biasing of some components to the user's personal preferences. Conventionally, contextual and personalized adaptation for language models, are achieved through class-based factorization, which requires class-annotated data, or through biasing to individual phrases which is limited in scale. In this paper, we propose a system that combines model-defined components, by learning when to activate the generation process from each individual component, and how to combine probability distributions from each component, directly from unlabeled text data.