Mohamed Khamis

Yaxiong Lei, Shijing He, Mohamed Khamis et al.

In recent years we have witnessed an increasing number of interactive systems on handheld mobile devices which utilise gaze as a single or complementary interaction modality. This trend is driven by the enhanced computational power of these devices, higher resolution and capacity of their cameras, and improved gaze estimation accuracy obtained from advanced machine learning techniques, especially in deep learning. As the literature is fast progressing, there is a pressing need to review the state of the art, delineate the boundary, and identify the key research challenges and opportunities in gaze estimation and interaction. This paper aims to serve this purpose by presenting an end-to-end holistic view in this area, from gaze capturing sensors, to gaze estimation workflows, to deep learning techniques, and to gaze interactive applications.

Yaxiong Lei, Yuheng Wang, Fergus Buchanan et al.

Mobile gaze tracking involves inferring a user's gaze point or direction on a mobile device's screen from facial images captured by the device's front camera. While this technology inspires an increasing number of gaze-interaction applications, achieving consistent accuracy remains challenging due to dynamic user-device spatial relationships and varied motion conditions inherent in mobile contexts. This paper provides empirical evidence on how user mobility and behaviour affect mobile gaze tracking accuracy. We conduct two user studies collecting behaviour and gaze data under various motion conditions - from lying to maze navigation - and during different interaction tasks. Quantitative analysis has revealed behavioural regularities among daily tasks and identified head distance, head pose, and device orientation as key factors affecting accuracy, with errors increasing by up to 48.91% in dynamic conditions compared to static ones. These findings highlight the need for more robust, adaptive eye-tracking systems that account for head movements and device deflection to maintain accuracy across diverse mobile contexts.

Yaxiong Lei, Mingyue Zhao, Yuheng Wang et al.

Mobile gaze tracking faces a fundamental challenge: maintaining accuracy as users naturally change their postures and device orientations. Traditional calibration approaches, like one-off, fail to adapt to these dynamic conditions, leading to degraded performance over time. We present MAC-Gaze, a Motion-Aware continual Calibration approach that leverages smartphone Inertial measurement unit (IMU) sensors and continual learning techniques to automatically detect changes in user motion states and update the gaze tracking model accordingly. Our system integrates a pre-trained visual gaze estimator and an IMU-based activity recognition model with a clustering-based hybrid decision-making mechanism that triggers recalibration when motion patterns deviate significantly from previously encountered states. To enable accumulative learning of new motion conditions while mitigating catastrophic forgetting, we employ replay-based continual learning, allowing the model to maintain performance across previously encountered motion conditions. We evaluate our system through extensive experiments on the publicly available RGBDGaze dataset and our own 10-hour multimodal MotionGaze dataset (481K+ images, 800K+ IMU readings), encompassing a wide range of postures under various motion conditions including sitting, standing, lying, and walking. Results demonstrate that our method reduces gaze estimation error by 19.9% on RGBDGaze (from 1.73 cm to 1.41 cm) and by 31.7% on MotionGaze (from 2.81 cm to 1.92 cm) compared to traditional calibration approaches. Our framework provides a robust solution for maintaining gaze estimation accuracy in mobile scenarios.

Mark McGill, Joseph O'Hagan, Thomas Goodge et al.

This position paper for the CHI 2025 workshop "Everyday AR through AI-in-the-Loop" reflects on whether as a field HCI needs to define Responsible XR as a parallel to, and in conjunction with, Responsible AI, addressing the unique vulnerabilities posed by mass adoption of wearable AI-enabled AR glasses and XR devices that could enact AI-driven human perceptual augmentation.

Wen-Jie Tseng, Elise Bonnail, Mark McGill et al.

"Virtual-Physical Perceptual Manipulations" (VPPMs) such as redirected walking and haptics expand the user's capacity to interact with Virtual Reality (VR) beyond what would ordinarily physically be possible. VPPMs leverage knowledge of the limits of human perception to effect changes in the user's physical movements, becoming able to (perceptibly and imperceptibly) nudge their physical actions to enhance interactivity in VR. We explore the risks posed by the malicious use of VPPMs. First, we define, conceptualize and demonstrate the existence of VPPMs. Next, using speculative design workshops, we explore and characterize the threats/risks posed, proposing mitigations and preventative recommendations against the malicious use of VPPMs. Finally, we implement two sample applications to demonstrate how existing VPPMs could be trivially subverted to create the potential for physical harm. This paper aims to raise awareness that the current way we apply and publish VPPMs can lead to malicious exploits of our perceptual vulnerabilities.

Yuanjie Xia, Mohamed Khamis, F. Anibal Fernandez et al.

Contact lenses have traditionally been used for vision correction applications. Recent advances in microelectronics and nanofabrication on flexible substrates have now enabled sensors, circuits and other essential components to be integrated on a small contact lens platform. This has opened up the possibility of using contact lenses for a range of human-machine interaction applications including vision assistance, eye tracking, displays and health care. In this article, we systematically review the range of smart contact lens materials, device architectures and components that facilitate this interaction for different applications. In fact, evidence from our systematic review demonstrates that these lenses can be used to display information, detect eye movements, restore vision and detect certain biomarkers in tear fluid. Consequently, whereas previous state-of the-art reviews in contact lenses focused exclusively on biosensing, our systematic review covers a wider range of smart contact lens applications in HMI. Moreover, we present a new method of classifying the literature on smart contact lenses according to their six constituent building blocks, which are the sensing, energy management, driver electronics, communications, substrate and the interfacing modules. Based on recent developments in each of these categories, we speculate the challenges and opportunities of smart contact lenses for human-machine interaction. Moreover, we propose a novel self-powered smart contact lens concept with integrated energy harvesters, sensors and communication modules to enable autonomous operation. Our review is therefore a critical evaluation of current data and is presented with the aim of guiding researchers to new research directions in smart contact lenses.

Christian Tiefenau, Maximilian Häring, Mohamed Khamis et al.

Nowadays, most mobile devices support biometric authentication schemes like fingerprint or face unlock. However, these probabilistic mechanisms can only be activated in combination with a second alternative factor, usually knowledge-based authentication. In this paper, we show that this aspect can be exploited in a bypass attack. In this bypass attack, the attacker forces the user to "bypass" the biometric authentication by, for example, resetting the phone. This forces the user to enter an easy-to-observe passcode instead. We present the threat model and provide preliminary results of an online survey. Based on our results, we discuss potential countermeasures. We conclude that better feedback design and security-optimized fallback mechanisms can help further improve the overall security of mobile unlock mechanisms while preserving usability.

Heiko Drewes, Mohamed Khamis, Florian Alt

We introduce and evaluate a novel approach for detecting smooth pursuit eye movements that increases the number of distinguishable targets and is more robust against false positives. Being natural and calibration-free, Pursuits has been gaining popularity in the past years. At the same time, current implementations show poor performance when more than eight on-screen targets are being used, thus limiting its applicability. Our approach (1) leverages the slope of a regression line, and (2) introduces a minimum signal duration that improves both the new and the traditional detection method. After introducing the approach as well as the implementation, we compare it to the traditional correlation-based Pursuits detection method. We tested the approach up to 24 targets and show that, if accepting a similar error rate, nearly twice as many targets can be distinguished compared to state of the art. For fewer targets, accuracy increases significantly. We believe our approach will enable more robust pursuit-based user interfaces, thus making it valuable for both researchers and practitioners.