Jie Yang

Yili Ren, Jie Yang

This paper presents GoPose, a 3D skeleton-based human pose estimation system that uses WiFi devices at home. Our system leverages the WiFi signals reflected off the human body for 3D pose estimation. In contrast to prior systems that need specialized hardware or dedicated sensors, our system does not require a user to wear or carry any sensors and can reuse the WiFi devices that already exist in a home environment for mass adoption. To realize such a system, we leverage the 2D AoA spectrum of the signals reflected from the human body and the deep learning techniques. In particular, the 2D AoA spectrum is proposed to locate different parts of the human body as well as to enable environment-independent pose estimation. Deep learning is incorporated to model the complex relationship between the 2D AoA spectrums and the 3D skeletons of the human body for pose tracking. Our evaluation results show GoPose achieves around 4.7cm of accuracy under various scenarios including tracking unseen activities and under NLoS scenarios.

Tianhe Ren, Shilong Liu, Ailing Zeng et al.

We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

Yili Ren, Jie Yang

WiFi human sensing has become increasingly attractive in enabling emerging human-computer interaction applications. The corresponding technique has gradually evolved from the classification of multiple activity types to more fine-grained tracking of 3D human poses. However, existing WiFi-based 3D human pose tracking is limited to a set of predefined activities. In this work, we present Winect, a 3D human pose tracking system for free-form activity using commodity WiFi devices. Our system tracks free-form activity by estimating a 3D skeleton pose that consists of a set of joints of the human body. In particular, we combine signal separation and joint movement modeling to achieve free-form activity tracking. Our system first identifies the moving limbs by leveraging the two-dimensional angle of arrival of the signals reflected off the human body and separates the entangled signals for each limb. Then, it tracks each limb and constructs a 3D skeleton of the body by modeling the inherent relationship between the movements of the limb and the corresponding joints. Our evaluation results show that Winect is environment-independent and achieves centimeter-level accuracy for free-form activity tracking under various challenging environments including the none-line-of-sight (NLoS) scenarios.

Zi Wang, Jie Yang

Biometric-based authentication is gaining increasing attention for wearables and mobile applications. Meanwhile, the growing adoption of sensors in wearables also provides opportunities to capture novel wearable biometrics. In this work, we propose EarDynamic, an ear canal deformation-based user authentication using in-ear wearables. EarDynamic provides continuous and passive user authentication and is transparent to users. It leverages ear canal deformation that combines the unique static geometry and dynamic motions of the ear canal when the user is speaking for authentication. It utilizes an acoustic sensing approach to capture the ear canal deformation with the built-in microphone and speaker of the in-ear wearable. Specifically, it first emits well-designed inaudible beep signals and records the reflected signals from the ear canal. It then analyzes the reflected signals and extracts fine-grained acoustic features that correspond to the ear canal deformation for user authentication. Our extensive experimental evaluation shows that EarDynamic can achieve a recall of 97.38% and an F1 score of 96.84%. Results also show that our system works well under different noisy environments with various daily activities.

Yili Ren, Jie Yang

The popularity of Internet-of-Things (IoT) has provided us with unprecedented opportunities to enable a variety of emerging services in a smart home environment. Among those services, sensing the liquid level in a container is critical to building many smart home and mobile healthcare applications that improve the quality of life. This paper presents LiquidSense, a liquid-level sensing system that is low-cost, high accuracy, widely applicable to different daily liquids and containers, and can be easily integrated with existing smart home networks. LiquidSense uses an existing home WiFi network and a low-cost transducer that attached to the container to sense the resonance of the container for liquid level detection. In particular, our system mounts a low-cost transducer on the surface of the container and emits a well-designed chirp signal to make the container resonant, which introduces subtle changes to the home WiFi signals. By analyzing the subtle phase changes of the WiFi signals, LiquidSense extracts the resonance frequency as a feature for liquid level detection. Our system constructs prediction models for both continuous and discrete predictions using curve fitting and SVM respectively. We evaluate LiquidSense in home environments with containers of three different materials and six types of liquids. Results show that LiquidSense achieves an overall accuracy of 97% for continuous prediction and an overall F-score of 0.968 for discrete prediction. Results also show that our system has a large coverage in a home environment and works well under non-line-of-sight (NLOS) scenarios.

Zi Wang, Jie Yang

Walking while using a smartphone is becoming a major pedestrian safety concern as people may unknowingly bump into various obstacles that could lead to severe injuries. In this paper, we propose ObstacleWatch, an acoustic-based obstacle collision detection system to improve the safety of pedestrians who are engaged in smartphone usage while walking. ObstacleWatch leverages the advanced audio hardware of the smartphone to sense the surrounding obstacles and infers fine-grained information about the frontal obstacle for collision detection. In particular, our system emits well-designed inaudible beep signals from the smartphone built-in speaker and listens to the reflections with the stereo recording of the smartphone. By analyzing the reflected signals received at two microphones, ObstacleWatch is able to extract fine-grained information of the frontal obstacle including the distance, angle, and size for detecting the possible collisions and to alert users. Our experimental evaluation under two real-world environments with different types of phones and obstacles shows that ObstacleWatch achieves over 92% accuracy in predicting obstacle collisions with distance estimation errors at about 2 cm. Results also show that ObstacleWatch is robust to different sizes of objects and is compatible with different phone models with low energy consumption.

Linghan Zhang, Jie Yang

Voice authentication is drawing increasing attention and becomes an attractive alternative to passwords for mobile authentication. Recent advances in mobile technology further accelerate the adoption of voice biometrics in an array of diverse mobile applications. However, recent studies show that voice authentication is vulnerable to replay attacks, where an adversary can spoof a voice authentication system using a pre-recorded voice sample collected from the victim. In this paper, we propose VoiceLive, a liveness detection system for both text-dependent and text-independent voice authentication on smartphones. VoiceLive detects a live user by leveraging the user's unique vocal system and the stereo recording of smartphones. In particular, utilizing the built-in gyroscope, loudspeaker, and microphone, VoiceLive first measures the smartphone's distance and angle from the user, then it captures the position-specific time-difference-of-arrival (TDoA) changes in a sequence of phoneme sounds to the two microphones of the phone, and uses such unique TDoA dynamic which doesn't exist under replay attacks for liveness detection. VoiceLive is practical as it doesn't require additional hardware but two-channel stereo recording that is supported by virtually all smartphones. Our experimental evaluation with 12 participants and different types of phones shows that VoiceLive achieves over 99% detection accuracy at around 1% Equal Error Rate (EER) on the text-dependent system and around 99% accuracy and 2% EER on the text-independent one. Results also show that VoiceLive is robust to different phone positions, i.e. the user is free to hold the smartphone with distinct distances and angles.

Sheng Tan, Jie Yang

This paper presents MultiTrack, a commodity WiFi-based human sensing system that can track multiple users and recognize the activities of multiple users performing them simultaneously. Such a system can enable easy and large-scale deployment for multi-user tracking and sensing without the need for additional sensors through the use of existing WiFi devices (e.g., desktops, laptops, and smart appliances). The basic idea is to identify and extract the signal reflection corresponding to each individual user with the help of multiple WiFi links and all the available WiFi channels at 5GHz. Given the extracted signal reflection of each user, MultiTrack examines the path of the reflected signals at multiple links to simultaneously track multiple users. It further reconstructs the signal profile of each user as if only a single user has performed activity in the environment to facilitate multi-user activity recognition. We evaluate MultiTrack in different multipath environments with up to 4 users for multi-user tracking and up to 3 users for activity recognition. Experimental results show that our system can achieve decimeter localization accuracy and over 92% activity recognition accuracy under multi-user scenarios.

Sheng Tan, Jie Yang

This paper presents FruitSense, a novel fruit ripeness sensing system that leverages wireless signals to enable non-destructive and low-cost detection of fruit ripeness. Such a system can reuse existing WiFi devices in homes without the need for additional sensors. It uses WiFi signals to sense the physiological changes associated with fruit ripening for detecting the ripeness of fruit. FruitSense leverages the larger bandwidth at 5GHz (i.e., over 600MHz) to extract the multipath-independent signal components to characterize the physiological compounds of the fruit. It then measures the similarity between the extracted features and the ones in ripeness profiles for identifying the ripeness level. We evaluate FruitSense in different multipath environments with two types of fruits (i.e, kiwi and avocado) under four levels of ripeness. Experimental results show that FruitSense can detect the ripeness levels of fruits with an accuracy of over 90%.

Linghan Zhang, Jie Yang

Voice biometrics is drawing increasing attention as it is a promising alternative to legacy passwords for user authentication. Recently, a growing body of work shows that voice biometrics is vulnerable to spoofing through replay attacks, where an adversary tries to spoof voice authentication systems by using a pre-recorded voice sample collected from a genuine user. To this end, we propose VoiceGesture, a liveness detection solution for voice authentication on smart devices such as smartphones and smart speakers. VoiceGesture detects a live user by leveraging both the unique articulatory gesture of the user when speaking a passphrase and the audio hardware advances on these smart devices. Specifically, our system re-uses a pair of built-in speaker and microphone on a smart device as a Doppler radar, which transmits a high-frequency acoustic sound from the speaker and listens to the reflections at the microphone when a user speaks a passphrase. Then we extract Doppler shifts resulted from the user's articulatory gestures for liveness detection. VoiceGesture is practical as it requires neither cumbersome operations nor additional hardware but a speaker and a microphone commonly available on smart devices that support voice input. Our experimental evaluation with 21 participants and different smart devices shows that VoiceGesture achieves over 99% and around 98% detection accuracy for text-dependent and text-independent liveness detection, respectively. Results also show that VoiceGesture is robust to different device placements, low audio sampling frequency, and supports medium range liveness detection on smart speakers in various use scenarios like smart homes and smart vehicles.

Sheng Tan, Jie Yang

Gesture recognition has become increasingly important in human-computer interaction and can support different applications such as smart home, VR, and gaming. Traditional approaches usually rely on dedicated sensors that are worn by the user or cameras that require line of sight. In this paper, we present fine-grained finger gesture recognition by using commodity WiFi without requiring user to wear any sensors. Our system takes advantages of the fine-grained Channel State Information available from commodity WiFi devices and the prevalence of WiFi network infrastructures. It senses and identifies subtle movements of finger gestures by examining the unique patterns exhibited in the detailed CSI. We devise environmental noise removal mechanism to mitigate the effect of signal dynamic due to the environment changes. Moreover, we propose to capture the intrinsic gesture behavior to deal with individual diversity and gesture inconsistency. Lastly, we utilize multiple WiFi links and larger bandwidth at 5GHz to achieve finger gesture recognition under multi-user scenario. Our experimental evaluation in different environments demonstrates that our system can achieve over 90% recognition accuracy and is robust to both environment changes and individual diversity. Results also show that our system can provide accurate gesture recognition under different scenarios.

Yang Hu, Yun Gu, Jie Yang et al.

In robotic surgery, task automation and learning from demonstration combined with human supervision is an emerging trend for many new surgical robot platforms. One such task is automated anastomosis, which requires bimanual needle handling and suture detection. Due to the complexity of the surgical environment and varying patient anatomies, reliable suture detection is difficult, which is further complicated by occlusion and thread topologies. In this paper, we propose a multi-stage framework for suture thread detection based on deep learning. Fully convolutional neural networks are used to obtain the initial detection and the overlapping status of suture thread, which are later fused with the original image to learn a gradient road map of the thread. Based on the gradient road map, multiple segments of the thread are extracted and linked to form the whole thread using a curvilinear structure detector. Experiments on two different types of sutures demonstrate the accuracy of the proposed framework.