On the Anisotropy of Score-Based Generative Models
This work provides a new tool for explaining and predicting directional biases in generative models, which is incremental but offers practical insights for researchers in machine learning.
The paper tackled the problem of understanding how network architecture influences the inductive biases of score-based generative models by introducing Score Anisotropy Directions (SADs), which reveal architecture-dependent directional preferences and correlate with generalization ability, as demonstrated through synthetic data and image benchmarks using Wasserstein metrics.
We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models.