D.Y. Yang

Deok-Kyeong Jang, Dongseok Yang, Deok-Yun Jang et al.

Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full-body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data-driven generative method for real-time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from LiDAR.As a central factor for high-accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state-of-the-art methods. Additionally, we implement a real-time application to showcase our method in real-world scenarios. MOVIN dataset is available at \url{https://movin3d.github.io/movin_pg2023/}.

Leonard Yoon, Dongseok Yang, Choongho Chung et al.

We present a novel mixed reality (MR) telepresence system enabling a local user to interact with a remote user through full-body avatars in their own rooms. If the remote rooms have different sizes and furniture arrangements, directly applying a user's motion to an avatar leads to a mismatch of placement and deictic gesture. To overcome this problem, we retarget the placement, arm gesture, and head movement of a local user to an avatar in a remote room to preserve a local user's environment and interaction context. This allows avatars to utilize real furniture and interact with a local user and shared objects as if they were in the same room. This paper describes our system's design and implementation in detail and a set of example scenarios in the living room and office room. A qualitative user study delves into a user experience, challenges, and possible extensions of the proposed system.

Leonard Yoon, Dongseok Yang, Jaehyun Kim et al.

Rapidly developing technologies are realizing a 3D telepresence, in which geographically separated users can interact with each other through their virtual avatars. In this paper, we present novel methods to determine the avatar's position in an indoor space to preserve the semantics of the user's position in a dissimilar indoor space with different space configurations and furniture layouts. To this end, we first perform a user survey on the preferred avatar placements for various indoor configurations and user placements, and identify a set of related attributes, including interpersonal relation, visual attention, pose, and spatial characteristics, and quantify these attributes with a set of features. By using the obtained dataset and identified features, we train a neural network that predicts the similarity between two placements. Next, we develop an avatar placement method that preserves the semantics of the placement of the remote user in a different space as much as possible. We show the effectiveness of our methods by implementing a prototype AR-based telepresence system and user evaluations.