Grid Partitioned Attention: Efficient TransformerApproximation with Inductive Bias for High Resolution Detail Generation
This addresses the problem of high computational and memory costs in attention-based models for image generation, particularly in conditional tasks like pose morphing, though it is incremental as it builds on existing attention methods with a sparse inductive bias.
The paper tackles the impractical memory requirements of attention mechanisms for high-resolution image generation by introducing Grid Partitioned Attention (GPA), an approximate algorithm that improves efficiency, leading to state-of-the-art results in human pose morphing benchmarks.
Attention is a general reasoning mechanism than can flexibly deal with image information, but its memory requirements had made it so far impractical for high resolution image generation. We present Grid Partitioned Attention (GPA), a new approximate attention algorithm that leverages a sparse inductive bias for higher computational and memory efficiency in image domains: queries attend only to few keys, spatially close queries attend to close keys due to correlations. Our paper introduces the new attention layer, analyzes its complexity and how the trade-off between memory usage and model power can be tuned by the hyper-parameters.We will show how such attention enables novel deep learning architectures with copying modules that are especially useful for conditional image generation tasks like pose morphing. Our contributions are (i) algorithm and code1of the novel GPA layer, (ii) a novel deep attention-copying architecture, and (iii) new state-of-the art experimental results in human pose morphing generation benchmarks.