Yan Long

Yan Long, Pirouz Naghavi, Blas Kojusner et al.

Our research discovers how the rolling shutter and movable lens structures widely found in smartphone cameras modulate structure-borne sounds onto camera images, creating a point-of-view (POV) optical-acoustic side channel for acoustic eavesdropping. The movement of smartphone camera hardware leaks acoustic information because images unwittingly modulate ambient sound as imperceptible distortions. Our experiments find that the side channel is further amplified by intrinsic behaviors of Complementary metal-oxide-semiconductor (CMOS) rolling shutters and movable lenses such as in Optical Image Stabilization (OIS) and Auto Focus (AF). Our paper characterizes the limits of acoustic information leakage caused by structure-borne sound that perturbs the POV of smartphone cameras. In contrast with traditional optical-acoustic eavesdropping on vibrating objects, this side channel requires no line of sight and no object within the camera's field of view (images of a ceiling suffice). Our experiments test the limits of this side channel with a novel signal processing pipeline that extracts and recognizes the leaked acoustic information. Our evaluation with 10 smartphones on a spoken digit dataset reports 80.66%, 91.28%, and 99.67% accuracies on recognizing 10 spoken digits, 20 speakers, and 2 genders respectively. We further systematically discuss the possible defense strategies and implementations. By modeling, measuring, and demonstrating the limits of acoustic eavesdropping from smartphone camera image streams, our contributions explain the physics-based causality and possible ways to reduce the threat on current and future devices.

Yan Long, Chen Yan, Shilin Xiao et al.

Using mathematical modeling and human subjects experiments, this research explores the extent to which emerging webcams might leak recognizable textual and graphical information gleaming from eyeglass reflections captured by webcams. The primary goal of our work is to measure, compute, and predict the factors, limits, and thresholds of recognizability as webcam technology evolves in the future. Our work explores and characterizes the viable threat models based on optical attacks using multi-frame super resolution techniques on sequences of video frames. Our models and experimental results in a controlled lab setting show it is possible to reconstruct and recognize with over 75% accuracy on-screen texts that have heights as small as 10 mm with a 720p webcam. We further apply this threat model to web textual contents with varying attacker capabilities to find thresholds at which text becomes recognizable. Our user study with 20 participants suggests present-day 720p webcams are sufficient for adversaries to reconstruct textual content on big-font websites. Our models further show that the evolution towards 4K cameras will tip the threshold of text leakage to reconstruction of most header texts on popular websites. Besides textual targets, a case study on recognizing a closed-world dataset of Alexa top 100 websites with 720p webcams shows a maximum recognition accuracy of 94% with 10 participants even without using machine-learning models. Our research proposes near-term mitigations including a software prototype that users can use to blur the eyeglass areas of their video streams. For possible long-term defenses, we advocate an individual reflection testing procedure to assess threats under various settings, and justify the importance of following the principle of least privilege for privacy-sensitive scenarios.

Yan Long, Kevin Fu

While the embedded security research community aims to protect systems by reducing analog sensor side channels, our work argues that sensor side channels can be beneficial to defenders. This work introduces the general problem of synthesizing virtual sensors from existing circuits to authenticate physical sensors' measurands. We investigate how to apply this approach and present a preliminary analytical framework and definitions for sensor side channels. To illustrate the general concept, we provide a proof-of-concept case study to synthesize a virtual inertial measurement unit from a camera motion side channel. Our work also provides an example of applying this technique to protect facial recognition against silicon mask spoofing attacks. Finally, we discuss downstream problems of how to ensure that side channels benefit the defender, but not the adversary, during authentication.

Connor Bolton, Yan Long, Jun Han et al.

Smartphone motion sensors provide a concealed mechanism for eavesdropping on acoustic information, like touchtones, emitted by a device. Eavesdropping on touchtones exposes credit card information, banking pins, and social security card numbers to malicious 3rd party apps requiring only motion sensor data. This paper's primary contribution is an analysis rooted in physics and signal processing theory of several eavesdropping mitigations, which could be implemented in a smartphone update. We verify our analysis imperially to show how previously suggested mitigations, i.e. a low-pass filter, can undesirably reduce the motion sensor data to all applications by 83% but only reduce an advanced adversary's accuracy by less than one percent. Other designs, i.e. anti-aliasing filters, can fully preserve the motion sensor data to support benign application functionality while reducing attack accuracy by 50.1%. We intend for this analysis to motivate the need for deployable mitigations against acoustic leakage on smartphone motion sensors, including but not limited to touchtones, while also providing a basis for future mitigations to improve upon.

Ke Wang, Gang Li, Junlan Chen et al.

China is the world's largest automotive market and is ambitious for autonomous vehicles (AVs) development. As one of the key goals of AVs, pedestrian safety is an important issue in China. Despite the rapid development of driverless technologies in recent years, there is a lack of researches on the adaptability of AVs to pedestrians. To fill the gap, this study would discuss the adaptability of current driverless technologies to China urban pedestrians by reviewing the latest researches. The paper firstly analyzed typical Chinese pedestrian behaviors and summarized the safety demands of pedestrians for AVs through articles and open database data, which are worked as the evaluation criteria. Then, corresponding driverless technologies are carefully reviewed. Finally, the adaptability would be given combining the above analyses. Our review found that autonomous vehicles have trouble in the occluded pedestrian environment and Chinese pedestrians do not accept AVs well. And more explorations should be conducted on standard human-machine interaction, interaction information overload avoidance, occluded pedestrians detection and nation-based receptivity research. The conclusions are very useful for motor corporations and driverless car researchers to place more attention on the complexity of the Chinese pedestrian environment, for transportation experts to protect pedestrian safety in the context of AVs, and for governors to think about making new pedestrians policies to welcome the upcoming driverless cars.