GameLabel-10K: Collecting Image Preference Data Through Mobile Game Crowdsourcing
This work offers a new, potentially cost-effective method for collecting large-scale image preference data for researchers and developers working with diffusion models, addressing the high data demands of large models.
This study explores a novel method for collecting image preference data by replacing paid annotators with mobile game players rewarded with in-game currency. They created GameLabel-10K, a dataset with nearly 10,000 labels and 7,000 unique prompts, and demonstrated its validity by fine-tuning Flux Schnell, which showed improved prompt adherence.
The rise of multi-billion parameter models has sparked an intense hunger for data across deep learning. This study explores the possibility of replacing paid annotators with video game players who are rewarded with in-game currency for good performance. We collaborate with the developers of a mobile historical strategy game, Armchair Commander, to test this idea. More specifically, the current study tests this idea using pairwise image preference data, typically used to fine-tune diffusion models. Using this method, we create GameLabel-10K, a dataset with slightly under 10 thousand labels and 7000 unique prompts. We fine-tune a model on this dataset, we fine-tune Flux Schnell and find an improvement in its prompt adherence, demonstrating the validity of our collection method. In addition, we publicly release both the dataset and our fine-tuned model on Hugging Face.