Bandit Structured Prediction for Neural Sequence-to-Sequence Learning
This addresses domain adaptation for neural machine translation, but it is incremental as it builds on existing bandit learning frameworks.
The paper tackles the problem of learning from partial feedback in neural sequence-to-sequence tasks by extending bandit structured prediction to attention-based RNNs, achieving up to 5.89 BLEU point improvements in neural machine translation domain adaptation.
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural networks. Furthermore, we show how to incorporate control variates into our learning algorithms for variance reduction and improved generalization. We present an evaluation on a neural machine translation task that shows improvements of up to 5.89 BLEU points for domain adaptation from simulated bandit feedback.