BattRAE: Bidimensional Attention-Based Recursive Autoencoders for Learning Bilingual Phrase Embeddings
This work addresses the challenge of improving bilingual phrase representations for machine translation, offering a novel method that enhances state-of-the-art systems, though it is incremental as it builds on existing recursive autoencoder and attention mechanisms.
The paper tackles the problem of learning bilingual phrase embeddings by proposing BattRAE, a bidimensional attention-based recursive autoencoder that integrates multi-granularity clues and source-target interactions, resulting in a substantial improvement of up to 1.63 BLEU points on average over a baseline in NIST Chinese-English machine translation.
In this paper, we propose a bidimensional attention based recursive autoencoder (BattRAE) to integrate clues and sourcetarget interactions at multiple levels of granularity into bilingual phrase representations. We employ recursive autoencoders to generate tree structures of phrases with embeddings at different levels of granularity (e.g., words, sub-phrases and phrases). Over these embeddings on the source and target side, we introduce a bidimensional attention network to learn their interactions encoded in a bidimensional attention matrix, from which we extract two soft attention weight distributions simultaneously. These weight distributions enable BattRAE to generate compositive phrase representations via convolution. Based on the learned phrase representations, we further use a bilinear neural model, trained via a max-margin method, to measure bilingual semantic similarity. To evaluate the effectiveness of BattRAE, we incorporate this semantic similarity as an additional feature into a state-of-the-art SMT system. Extensive experiments on NIST Chinese-English test sets show that our model achieves a substantial improvement of up to 1.63 BLEU points on average over the baseline.