CountSteer: Steering Attention for Object Counting in Diffusion Models
This addresses the gap between language and visual representation in diffusion models for more controllable text-to-image generation, but it is incremental as it builds on existing models without major architectural changes.
The paper tackled the problem of text-to-image diffusion models failing to follow numerical instructions by introducing CountSteer, a training-free method that steers cross-attention hidden states during inference, improving object-count accuracy by about 4% without compromising visual quality.
Text-to-image diffusion models generate realistic and coherent images but often fail to follow numerical instructions in text, revealing a gap between language and visual representation. Interestingly, we found that these models are not entirely blind to numbers-they are implicitly aware of their own counting accuracy, as their internal signals shift in consistent ways depending on whether the output meets the specified count. This observation suggests that the model already encodes a latent notion of numerical correctness, which can be harnessed to guide generation more precisely. Building on this intuition, we introduce CountSteer, a training-free method that improves generation of specified object counts by steering the model's cross-attention hidden states during inference. In our experiments, CountSteer improved object-count accuracy by about 4% without compromising visual quality, demonstrating a simple yet effective step toward more controllable and semantically reliable text-to-image generation.