TC-LoRA: Temporally Modulated Conditional LoRA for Adaptive Diffusion Control
This addresses the limitation of fixed architectures in diffusion models for adaptive control, offering a novel paradigm for dynamic conditioning in generative tasks.
The paper tackles the problem of static conditioning in controllable diffusion models by introducing TC-LoRA, a method that dynamically adapts model weights at each diffusion step, resulting in significant improvements in generative fidelity and adherence to spatial conditions compared to existing methods.
Current controllable diffusion models typically rely on fixed architectures that modify intermediate activations to inject guidance conditioned on a new modality. This approach uses a static conditioning strategy for a dynamic, multi-stage denoising process, limiting the model's ability to adapt its response as the generation evolves from coarse structure to fine detail. We introduce TC-LoRA (Temporally Modulated Conditional LoRA), a new paradigm that enables dynamic, context-aware control by conditioning the model's weights directly. Our framework uses a hypernetwork to generate LoRA adapters on-the-fly, tailoring weight modifications for the frozen backbone at each diffusion step based on time and the user's condition. This mechanism enables the model to learn and execute an explicit, adaptive strategy for applying conditional guidance throughout the entire generation process. Through experiments on various data domains, we demonstrate that this dynamic, parametric control significantly enhances generative fidelity and adherence to spatial conditions compared to static, activation-based methods. TC-LoRA establishes an alternative approach in which the model's conditioning strategy is modified through a deeper functional adaptation of its weights, allowing control to align with the dynamic demands of the task and generative stage.