Hee-Seung Moon

Xuejing Luo, Hee-Seung Moon, Christian Holz et al.

Selecting out-of-reach objects is a fundamental task in mixed reality (MR). Existing methods rely on a single cue or deterministically fuse multiple cues, leading to performance degradation when the dominant cue becomes unreliable. In this work, we introduce a probabilistic cue integration framework that enables flexible combination of multiple user-generated cues for intent inference. Inspired by natural grasping behavior, we instantiate the framework with pointing direction and grasp gestures as a new interaction technique, Point&Grasp. To this end, we collect the Out-of-Reach Grasping (ORG) dataset to train a robust likelihood model of the gestural cue, which captures grasping patterns not present in existing in-reach datasets. User studies demonstrate that our selection method with cue integration not only improves accuracy and speed over single-cue baselines, but also remains practically effective compared to state-of-the-art methods across various sources of ambiguity. The dataset and code are available at https://github.com/drlxj/point-and-grasp.

Hee-Seung Moon, Jiwon Seo

Accurate prediction of human movements is required to enhance the efficiency of physical human-robot interaction. Behavioral differences across various users are crucial factors that limit the prediction of human motion. Although recent neural network-based modeling methods have improved their prediction accuracy, most did not consider an effective adaptations to different users, thereby employing the same model parameters for all users. To deal with this insufficiently addressed challenge, we introduce a meta-learning framework to facilitate the rapid adaptation of the model to unseen users. In this study, we propose a model structure and a meta-learning algorithm specialized to enable fast user adaptation in predicting human movements in cooperative situations with robots. The proposed prediction model comprises shared and adaptive parameters, each addressing the user's general and individual movements. Using only a small amount of data from an individual user, the adaptive parameters are adjusted to enable user-specific prediction through a two-step process: initialization via a separate network and adaptation via a few gradient steps. Regarding the motion dataset that has 20 users collaborating with a robotic device, the proposed method outperforms existing meta-learning and non-meta-learning baselines in predicting the movements of unseen users.

Hee-Seung Moon, Jiwon Seo

The recently advanced robotics technology enables robots to assist users in their daily lives. Haptic guidance (HG) improves users' task performance through physical interaction between robots and users. It can be classified into optimal action-based HG (OAHG), which assists users with an optimal action, and user prediction-based HG (UPHG), which assists users with their next predicted action. This study aims to understand the difference between OAHG and UPHG and propose a combined HG (CombHG) that achieves optimal performance by complementing each HG type, which has important implications for HG design. We propose implementation methods for each HG type using deep learning-based approaches. A user study (n=20) in a haptic task environment indicated that UPHG induces better subjective evaluations, such as naturalness and comfort, than OAHG. In addition, the CombHG that we proposed further decreases the disagreement between the user intention and HG, without reducing the objective and subjective scores.

Hee-Seung Moon, Jiwon Seo

Training a robot that engages with people is challenging; it is expensive to directly involve people in the training process, which requires numerous data samples. This paper presents an alternative approach for resolving this problem. We propose a human path prediction network (HPPN) that generates a user's future trajectory based on sequential robot actions and human responses using a recurrent-neural-network structure. Subsequently, an evolution-strategy-based robot training method using only the virtual human movements generated using the HPPN is presented. It is demonstrated that our proposed method permits sample-efficient training of a robotic guide for visually impaired people. By collecting only 1.5 K episodes from real users, we were able to train the HPPN and generate more than 100 K virtual episodes required for training the robot. The trained robot precisely guided blindfolded participants along a target path. Furthermore, using virtual episodes, we investigated a new reward design that prioritizes human comfort during the robot's guidance without incurring additional costs. This sample-efficient training method is expected to be widely applicable to future robots that interact physically with humans.

Hee-Seung Moon, Jiwon Seo

Dynamic difficulty adjustment (DDA) is a technology that adapts a game's challenge to match the player's skill. It is a key element in game development that provides continuous motivation and immersion to the player. However, conventional DDA methods require tuning in-game parameters to generate the levels for various players. Recent DDA approaches based on deep learning can shorten the time-consuming tuning process, but require sufficient user demo data for adaptation. In this paper, we present a fast user adaptation method that can adjust the difficulty of the game for various players using only a small amount of demo data by applying a meta-learning algorithm. In the video game environment user test (n=9), our proposed DDA method outperformed a typical deep learning-based baseline method.

Hee-Seung Moon, Jiwon Seo

Social intelligence is an important requirement for enabling robots to collaborate with people. In particular, human path prediction is an essential capability for robots in that it prevents potential collision with a human and allows the robot to safely make larger movements. In this paper, we present a method for predicting the trajectory of a human who follows a haptic robotic guide without using sight, which is valuable for assistive robots that aid the visually impaired. We apply a deep learning method based on recurrent neural networks using multimodal data: (1) human trajectory, (2) movement of the robotic guide, (3) haptic input data measured from the physical interaction between the human and the robot, (4) human depth data. We collected actual human trajectory and multimodal response data through indoor experiments. Our model outperformed the baseline result while using only the robot data with the observed human trajectory, and it shows even better results when using additional haptic and depth data.