Are BLEU and Meaning Representation in Opposition?
This work addresses a fundamental trade-off in NLP for researchers and practitioners, revealing that optimizing for translation metrics may degrade representation utility, which is incremental as it builds on existing attention-based NMT architectures.
The paper investigates the relationship between translation quality and sentence representation quality in neural machine translation systems, finding that higher BLEU scores correlate with worse performance of the learned representations on classification and similarity tasks.
One of possible ways of obtaining continuous-space sentence representations is by training neural machine translation (NMT) systems. The recent attention mechanism however removes the single point in the neural network from which the source sentence representation can be extracted. We propose several variations of the attentive NMT architecture bringing this meeting point back. Empirical evaluation suggests that the better the translation quality, the worse the learned sentence representations serve in a wide range of classification and similarity tasks.