LMPriors: Pre-Trained Language Models as Task-Specific Priors
This addresses the problem of enhancing model learning with suitable priors for machine learning practitioners, though it is incremental as it builds on existing language model successes.
The paper tackles the challenge of incorporating task-specific priors in low-data regimes by using pre-trained language models to distill common-sense reasoning from natural language metadata, resulting in improved performance on tasks like feature selection, causal inference, and safe reinforcement learning.
Particularly in low-data regimes, an outstanding challenge in machine learning is developing principled techniques for augmenting our models with suitable priors. This is to encourage them to learn in ways that are compatible with our understanding of the world. But in contrast to generic priors such as shrinkage or sparsity, we draw inspiration from the recent successes of large-scale language models (LMs) to construct task-specific priors distilled from the rich knowledge of LMs. Our method, Language Model Priors (LMPriors), incorporates auxiliary natural language metadata about the task -- such as variable names and descriptions -- to encourage downstream model outputs to be consistent with the LM's common-sense reasoning based on the metadata. Empirically, we demonstrate that LMPriors improve model performance in settings where such natural language descriptions are available, and perform well on several tasks that benefit from such prior knowledge, such as feature selection, causal inference, and safe reinforcement learning.