Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
This provides an open resource and improved evaluation metric for text-to-image generation researchers, addressing a bottleneck previously limited to companies.
The authors tackled the lack of public human preference datasets for text-to-image generation by creating Pick-a-Pic, an open dataset from user inputs, and used it to train PickScore, a CLIP-based scoring function that achieves superhuman performance in predicting preferences and outperforms other metrics in model evaluation.
The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.