Text-to-Image Generation Via Energy-Based CLIP
This work addresses a bottleneck in multimodal AI by improving text-to-image generation, though it is incremental as it builds on existing JEM and CLIP frameworks.
The paper tackles scaling Joint Energy Models (JEMs) to high-resolution datasets by introducing CLIP-JEM, which integrates generative and discriminative objectives using CLIP, resulting in competitive performance on benchmarks with fewer parameters and enhanced guidance capabilities.
Joint Energy Models (JEMs), while drawing significant research attention, have not been successfully scaled to real-world, high-resolution datasets. We present CLIP-JEM, a novel approach extending JEMs to the multimodal vision-language domain using CLIP, integrating both generative and discriminative objectives. For the generative one, we introduce an image-text joint-energy function based on Cosine similarity in the CLIP space, training CLIP to assign low energy to real image-caption pairs and high energy otherwise. For the discriminative one, we employ contrastive adversarial loss, extending the adversarial training objective to the multimodal domain. CLIP-JEM not only generates realistic images from text but also achieves competitive results on the compositionality benchmark, outperforming leading methods with fewer parameters. Additionally, we demonstrate the superior guidance capability of CLIP-JEM by enhancing CLIP-based generative frameworks and converting unconditional diffusion models to text-based ones. Lastly, we show that our model can serve as a more robust evaluation metric for text-to-image generative tasks than CLIP.