UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models
This work addresses computational bottlenecks for researchers and practitioners training control adapters on diffusion models, though it appears incremental as it builds on existing adapter methods like ControlNets.
The paper tackles the problem of inefficient training for control adapters in large-scale diffusion models by introducing UniCon, which uses unidirectional information flow to eliminate gradient computation in the diffusion model during adapter training. This approach reduces GPU memory usage by one-third and increases training speed by 2.3 times while maintaining adapter performance.
We introduce UniCon, a novel architecture designed to enhance control and efficiency in training adapters for large-scale diffusion models. Unlike existing methods that rely on bidirectional interaction between the diffusion model and control adapter, UniCon implements a unidirectional flow from the diffusion network to the adapter, allowing the adapter alone to generate the final output. UniCon reduces computational demands by eliminating the need for the diffusion model to compute and store gradients during adapter training. Our results indicate that UniCon reduces GPU memory usage by one-third and increases training speed by 2.3 times, while maintaining the same adapter parameter size. Additionally, without requiring extra computational resources, UniCon enables the training of adapters with double the parameter volume of existing ControlNets. In a series of image conditional generation tasks, UniCon has demonstrated precise responsiveness to control inputs and exceptional generation capabilities.