ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation
This work addresses performance bottlenecks in sequence generation for NLP applications, offering a novel method that is incremental but shows strong specific gains.
The paper tackled the problem of improving sequence generation tasks like machine translation by proposing ODE Transformer, a new architecture inspired by numerical ODE methods, which achieved significant performance gains, such as 30.77 and 44.11 BLEU scores on WMT'14 benchmarks, with a slight inference efficiency cost.
Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, {\it ODE Transformer}, which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency.