Paired Examples as Indirect Supervision in Latent Decision Models
This work addresses the problem of weak supervision in latent decision models for researchers in compositional AI, offering an incremental improvement through indirect supervision techniques.
The paper tackles the challenge of learning compositional models where end-task supervision is weak, by introducing a method that uses paired examples to encourage consistency in latent decisions without requiring external supervision. The approach improves compositional question answering on the DROP dataset, enhancing in- and out-of-distribution generalization and leading to correct latent decision predictions.
Compositional, structured models are appealing because they explicitly decompose problems and provide interpretable intermediate outputs that give confidence that the model is not simply latching onto data artifacts. Learning these models is challenging, however, because end-task supervision only provides a weak indirect signal on what values the latent decisions should take. This often results in the model failing to learn to perform the intermediate tasks correctly. In this work, we introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. When two related training examples share internal substructure, we add an additional training objective to encourage consistency between their latent decisions. Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves. We apply our method to improve compositional question answering using neural module networks on the DROP dataset. We explore three ways to acquire paired questions in DROP: (a) discovering naturally occurring paired examples within the dataset, (b) constructing paired examples using templates, and (c) generating paired examples using a question generation model. We empirically demonstrate that our proposed approach improves both in- and out-of-distribution generalization and leads to correct latent decision predictions.