Latent Diffusion U-Net Representations Contain Positional Embeddings and Anomalies
This work highlights potential issues in using diffusion model representations for downstream tasks, which is incremental as it builds on existing analysis of model properties.
The paper analyzed Stable Diffusion models to understand the robustness of their representations, revealing learned positional embeddings, corner artifacts, and anomalous high-norm artifacts in intermediate layers.
Diffusion models have demonstrated remarkable capabilities in synthesizing realistic images, spurring interest in using their representations for various downstream tasks. To better understand the robustness of these representations, we analyze popular Stable Diffusion models using representational similarity and norms. Our findings reveal three phenomena: (1) the presence of a learned positional embedding in intermediate representations, (2) high-similarity corner artifacts, and (3) anomalous high-norm artifacts. These findings underscore the need to further investigate the properties of diffusion model representations before considering them for downstream tasks that require robust features. Project page: https://jonasloos.github.io/sd-representation-anomalies