Reified Context Models
This addresses a fundamental problem in probabilistic modeling for natural language processing, offering a novel way to balance accuracy and coverage, though it appears incremental in its application.
The paper tackles the trade-off between exact inference in simple models and approximate inference in complex models by introducing reified context models, which allow dynamic context selection at run-time, achieving expressivity and coverage on three natural language tasks.
A classic tension exists between exact inference in a simple model and approximate inference in a complex model. The latter offers expressivity and thus accuracy, but the former provides coverage of the space, an important property for confidence estimation and learning with indirect supervision. In this work, we introduce a new approach, reified context models, to reconcile this tension. Specifically, we let the amount of context (the arity of the factors in a graphical model) be chosen "at run-time" by reifying it---that is, letting this choice itself be a random variable inside the model. Empirically, we show that our approach obtains expressivity and coverage on three natural language tasks.