On the Limits of Latent Reuse in Diffusion Models
Provides theoretical guidance for practitioners deciding whether to reuse latent spaces in diffusion models for shifted datasets.
This paper theoretically analyzes when latent reuse in diffusion models is reliable under distribution shift, showing that score error depends on subspace misalignment and noise amplification. It also characterizes the required latent dimension for mixed training.
Diffusion models are often trained in low-dimensional latent spaces, which are then reused for related but shifted datasets. In this work, we study when such latent reuse remains reliable under distribution shift. We consider a source-target setting in which both datasets are approximately low-dimensional but may lie near different subspaces. We show that freezing and reusing a source latent space induces a target-domain score error governed by two quantities: the principal-angle misalignment between the source and target subspaces, and the target ambient noise amplified by the diffusion time scale. Motivated by these limits, we further study mixed source-target training and characterize how the required shared latent dimension depends on the relative geometry of the two distributions. Our results provide theoretical guidance on when latent reuse is reliable and when learning a shared representation may be necessary.