Shuling Zhao

Shuling Zhao, Fa-Ting Hong, Xiaoshui Huang et al.

Talking head video generation aims to generate a realistic talking head video that preserves the person's identity from a source image and the motion from a driving video. Despite the promising progress made in the field, it remains a challenging and critical problem to generate videos with accurate poses and fine-grained facial details simultaneously. Essentially, facial motion is often highly complex to model precisely, and the one-shot source face image cannot provide sufficient appearance guidance during generation due to dynamic pose changes. To tackle the problem, we propose to jointly learn motion and appearance codebooks and perform multi-scale codebook compensation to effectively refine both the facial motion conditions and appearance features for talking face image decoding. Specifically, the designed multi-scale motion and appearance codebooks are learned simultaneously in a unified framework to store representative global facial motion flow and appearance patterns. Then, we present a novel multi-scale motion and appearance compensation module, which utilizes a transformer-based codebook retrieval strategy to query complementary information from the two codebooks for joint motion and appearance compensation. The entire process produces motion flows of greater flexibility and appearance features with fewer distortions across different scales, resulting in a high-quality talking head video generation framework. Extensive experiments on various benchmarks validate the effectiveness of our approach and demonstrate superior generation results from both qualitative and quantitative perspectives when compared to state-of-the-art competitors.

Shuling Zhao, Dan Xu

Building 3D animatable head avatars from a single image is an important yet challenging problem. Existing methods generally collapse under large camera pose variations, compromising the realism of 3D avatars. In this work, we propose a new framework to tackle the novel setting of one-shot 3D full-head animatable avatar reconstruction in a single feed-forward pass, enabling real-time animation and simultaneous 360$^\circ$ rendering views. To facilitate efficient animation control, we model 3D head avatars with Gaussian primitives embedded on the surface of a parametric face model within the UV space. To obtain knowledge of full-head geometry and textures, we leverage rich 3D full-head priors within a pretrained 3D generative adversarial network (GAN) for global full-head feature extraction and multi-view supervision. To increase the fidelity of the 3D reconstruction of the input image, we take advantage of the symmetric nature of the UV space and human faces to fuse local fine-grained input image features with the global full-head textures. Extensive experiments demonstrate the effectiveness of our method, achieving high-quality 3D full-head modeling as well as real-time animation, thereby improving the realism of 3D talking avatars.

Hancheng Cao, Zhilong Chen, Mengjie Cheng et al.

As an emerging business phenomenon especially in China, instant messaging (IM) based social commerce is growing increasingly popular, attracting hundreds of millions of users and is becoming one important way where people make everyday purchases. Such platforms embed shopping experiences within IM apps, e.g., WeChat, WhatsApp, where real-world friends post and recommend products from the platforms in IM group chats and quite often form lasting recommending/buying relationships. How and why do users engage in IM based social commerce? Do such platforms create novel experiences that are distinct from prior commerce? And do these platforms bring changes to user social lives and relationships? To shed light on these questions, we launched a qualitative study where we carried out semi-structured interviews on 12 instant messaging based social commerce users in China. We showed that IM based social commerce: 1) enables more reachable, cost-reducing, and immersive user shopping experience, 2) shapes user decision-making process in shopping through pre-existing social relationship, mutual trust, shared identity, and community norm, and 3) creates novel social interactions, which can contribute to new tie formation while maintaining existing social relationships. We demonstrate that all these unique aspects link closely to the characteristics of IM platforms, as well as the coupling of user social and economic lives under such business model. Our study provides important research and design implications for social commerce, and decentralized, trusted socio-technical systems in general.