FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation
This addresses a persistent problem in image generation for applications requiring realistic hand synthesis, though it is incremental as it builds on existing diffusion models with domain-specific data.
The paper tackles the challenge of generating realistic hand images by introducing FoundHand, a large-scale domain-specific diffusion model that uses 2D keypoints for control, achieving state-of-the-art performance in tasks like reposing hands and fixing malformed images.
Despite remarkable progress in image generation models, generating realistic hands remains a persistent challenge due to their complex articulation, varying viewpoints, and frequent occlusions. We present FoundHand, a large-scale domain-specific diffusion model for synthesizing single and dual hand images. To train our model, we introduce FoundHand-10M, a large-scale hand dataset with 2D keypoints and segmentation mask annotations. Our insight is to use 2D hand keypoints as a universal representation that encodes both hand articulation and camera viewpoint. FoundHand learns from image pairs to capture physically plausible hand articulations, natively enables precise control through 2D keypoints, and supports appearance control. Our model exhibits core capabilities that include the ability to repose hands, transfer hand appearance, and even synthesize novel views. This leads to zero-shot capabilities for fixing malformed hands in previously generated images, or synthesizing hand video sequences. We present extensive experiments and evaluations that demonstrate state-of-the-art performance of our method.