Scalable Adaptive Computation for Iterative Generation
This addresses scalability and efficiency challenges in generative AI for domains like image and video synthesis, though it is an incremental improvement building on existing attention-based methods.
The paper tackles the inefficiency of uniform computation in architectures for high-dimensional data generation by proposing Recurrent Interface Networks (RINs), which decouple core computation from data dimensionality using latent tokens and cross-attention, resulting in state-of-the-art image and video generation up to 1024x1024 resolution with up to 10x efficiency gains over U-Nets.
Natural data is redundant yet predominant architectures tile computation uniformly across their input and output space. We propose the Recurrent Interface Networks (RINs), an attention-based architecture that decouples its core computation from the dimensionality of the data, enabling adaptive computation for more scalable generation of high-dimensional data. RINs focus the bulk of computation (i.e. global self-attention) on a set of latent tokens, using cross-attention to read and write (i.e. route) information between latent and data tokens. Stacking RIN blocks allows bottom-up (data to latent) and top-down (latent to data) feedback, leading to deeper and more expressive routing. While this routing introduces challenges, this is less problematic in recurrent computation settings where the task (and routing problem) changes gradually, such as iterative generation with diffusion models. We show how to leverage recurrence by conditioning the latent tokens at each forward pass of the reverse diffusion process with those from prior computation, i.e. latent self-conditioning. RINs yield state-of-the-art pixel diffusion models for image and video generation, scaling to 1024X1024 images without cascades or guidance, while being domain-agnostic and up to 10X more efficient than 2D and 3D U-Nets.