UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion
This work addresses the problem of scalable composite conditioning in diffusion models, offering a memory-efficient alternative to training separate adapters for each modality.
UNITY introduces a two-stage training paradigm for diffusion-based image generation that jointly learns shared semantics across multiple conditioning modalities before specializing, achieving state-of-the-art image fidelity with reduced inference latency and memory consumption.
We introduce UNITY, a Universal-to-Specialized adapter for efficient and scalable composite conditioning in diffusion based image generation. Unlike prior methods that train separate adapters for each conditioning modality, UNITY jointly learns shared semantics across multiple conditioning types and subsequently specializes without modifying the underlying architecture. The proposed two stage training paradigm consists of a Universal Stage that captures cross modal representations across all conditioning modalities using half of the total training steps, followed by a Specialization Stage that refines modality specific features using the remaining training budget. At the core of UNITY are the Morphable Attention Flow (MAF) Network and Morph Wrapper modules, which enable channel aware and spatially adaptive feature alignment through learnable flow fields and attention based fusion. This constant complexity formulation supports flexible operation under both single and composite conditioning settings while significantly reducing inference latency and memory consumption. Extensive experiments across multiple datasets demonstrate that UNITY achieves state of the art image fidelity while maintaining superior memory efficiency. Code: https://github.com/arya-domain/UNITY