Twin Networks: Matching the Future for Sequence Generation
This addresses the challenge of planning ahead in sequence generation for tasks like speech recognition and image captioning, offering an incremental improvement over existing methods.
The paper tackles the problem of long-term dependencies in generative RNNs by proposing a technique that uses a backward network during training to encourage forward states to predict future information, resulting in a 9% relative improvement in speech recognition and significant gains in COCO caption generation.
We propose a simple technique for encouraging generative RNNs to plan ahead. We train a "backward" recurrent network to generate a given sequence in reverse order, and we encourage states of the forward model to predict cotemporal states of the backward model. The backward network is used only during training, and plays no role during sampling or inference. We hypothesize that our approach eases modeling of long-term dependencies by implicitly forcing the forward states to hold information about the longer-term future (as contained in the backward states). We show empirically that our approach achieves 9% relative improvement for a speech recognition task, and achieves significant improvement on a COCO caption generation task.