EEL: Efficiently Encoding Lattices for Reranking
This work addresses a computational bottleneck in reranking for text generation, offering a practical speedup for researchers and practitioners, though it is incremental as it builds on existing lattice and Transformer techniques.
The paper tackles the inefficiency of reranking text generation hypotheses using pre-trained language models by introducing EEL, a method that encodes entire lattices of outputs in a single Transformer pass, achieving substantial speedup with minimal performance degradation across three tasks.
Standard decoding approaches for conditional text generation tasks typically search for an output hypothesis with high model probability, but this may not yield the best hypothesis according to human judgments of quality. Reranking to optimize for "downstream" metrics can better optimize for quality, but many metrics of interest are computed with pre-trained language models, which are slow to apply to large numbers of hypotheses. We explore an approach for reranking hypotheses by using Transformers to efficiently encode lattices of generated outputs, a method we call EEL. With a single Transformer pass over the entire lattice, we can approximately compute a contextualized representation of each token as if it were only part of a single hypothesis in isolation. We combine this approach with a new class of token-factored rerankers (TFRs) that allow for efficient extraction of high reranker-scoring hypotheses from the lattice. Empirically, our approach incurs minimal degradation error compared to the exponentially slower approach of encoding each hypothesis individually. When applying EEL with TFRs across three text generation tasks, our results show both substantial speedup compared to naive reranking and often better performance on downstream metrics than comparable approaches.