Jérémy Frey

Jérémy Frey, Gilad Ostrin, May Grabli et al.

We put forth Physiologically Driven Storytelling, a new approach to interactive storytelling where narratives adaptively unfold based on the reader's physiological state. We first describe a taxonomy framing how physiological signals can be used to drive interactive systems both as input and output. We then propose applications to interactive storytelling and describe the implementation of a software tool to create Physiological Interactive Fiction (PIF). The results of an online study (N=140) provided guidelines towards augmenting the reading experience. PIF was then evaluated in a lab study (N=14) to determine how physiological signals can be used to infer a reader's state. Our results show that breathing, electrodermal activity, and eye tracking can help differentiate positive from negative tones, and monotonous from exciting events. This work demonstrates how PIF can support storytelling in creating engaging content and experience tailored to the reader. Moreover, it opens the space to future physiologically driven systems within broader application areas.

Gilad Ostrin, Jérémy Frey, Jessica Cauchard

Interactive fiction is a literary genre that is rapidly gaining popularity. In this genre, readers are able to explicitly take actions in order to guide the course of the story. With the recent popularity of narrative focused games, we propose to design and develop an interactive narrative tool for content creators. In this extended abstract, we show how we leverage this interactive medium to present a tool for interactive storytelling in virtual reality. Using a simple markup language, content creators and researchers are now able to create interactive narratives in a virtual reality environment. We further discuss the potential future directions for a virtual reality storytelling engine.

Jérémy Frey, Jessica Cauchard

Biofeedback is commonly used to regulate one's state, for example to manage stress. The underlying idea is that by perceiving a feedback about their physiological activity, a user can act upon it. In this paper we describe through two recent projects how biofeedback could be leveraged to share one's state at distance. Such extension of biofeedback could answer to the need of belonging, further widening the applications of the technology in terms of well-being.

Morgane Hamon, Rémy Ramadour, Jérémy Frey

Anxiety is a common health issue that can occur throughout one's existence. In this pilot study we explore an alternative technique to regulate it: biofeedback. The long-term objective is to offer an ecological device that could help people cope with anxiety, by exposing their inner state in a comprehensive manner. We propose a first iteration of this device, "Inner Flower", that uses heart rate to adapt a breathing guide to the user, and we investigate its efficiency and usability. Traditionally, such device requires user's full attention. We propose an ambient modality during which the device operates in the peripheral vision. Beside comparing "Ambient" and "Focus" conditions, we also compare the biofeedback with a sham feedback (fixed breathing guide). We found that the Focus group demonstrated higher relaxation and performance on a cognitive task (N-back). However, there was no noticeable effect of the Ambient feedback, and the biofeedback condition did not yield any significant difference when compared to the sham feedback. These results, while promising, highlight the pitfalls of any research related to biofeedback, where it is difficult to fully comprehend the underlying mechanisms of such technique.

Jelena Mladenović, Jérémy Frey, Emmanuel Maby et al.

Adaptive Brain-Computer interfaces (BCIs) have shown to improve performance, however a general and flexible framework to implement adaptive features is still lacking. We appeal to a generic Bayesian approach, called Active Inference (AI), to infer user's intentions or states and act in a way that optimizes performance. In realistic P300-speller simulations, AI outperforms traditional algorithms with an increase in bit rate between 18% and 59%, while offering a possibility of unifying various adaptive implementations within one generic framework.

Emmanuel Christophe, Jérémy Frey, Richard Kronland-Martinet et al.

Designing a feedback that helps participants to achieve higher performances is an important concern in brain-computer interface (BCI) research. In a pilot study, we demonstrate how a congruent auditory feedback could improve classification in a electroencephalography (EEG) motor imagery BCI. This is a promising result for creating alternate feedback modality.

Jelena Mladenovic, Jérémy Frey, Jessica Cauchard

We present a system that raises awareness about users' inner state. Dišimo is a multimodal ambient display that provides feedback about one's stress level, which is assessed through heart rate monitoring. Upon detecting a low heart rate variability for a prolonged period of time, Dišimo plays an audio track, setting the pace of a regular and deep breathing. Users can then choose to take a moment to focus on their breath. By doing so, they will activate the Dišimo devices belonging to their close ones, who can then join for a shared relaxation session.

Jérémy Frey, May Grabli, Ronit Slyper et al.

Digitally presenting physiological signals as biofeedback to users raises awareness of both body and mind. This paper describes the effectiveness of conveying a physiological signal often overlooked for communication: breathing. We present the design and development of digital breathing patterns and their evaluation along three output modalities: visual, audio, and haptic. We also present Breeze, a wearable pendant placed around the neck that measures breathing and sends biofeedback in real-time. We evaluated how the breathing patterns were interpreted in a fixed environment and gathered qualitative data on the wearable device's design. We found that participants intentionally modified their own breathing to match the biofeedback, as a technique for understanding the underlying emotion. Our results describe how the features of the breathing patterns and the feedback modalities influenced participants' perception. We include guidelines and suggested use cases, such as Breeze being used by loved ones to increase connectedness and empathy.

Jelena Mladenović, Jérémy Frey, Manon Bonnet-Save et al.

Major issues in Brain Computer Interfaces (BCIs) include low usability and poor user performance. This paper tackles them by ensuring the users to be in a state of immersion, control and motivation, called state of flow. Indeed, in various disciplines, being in the state of flow was shown to improve performances and learning. Hence, we intended to draw BCI users in a flow state to improve both their subjective experience and their performances. In a Motor Imagery BCI game, we manipulated flow in two ways: 1) by adapting the task difficulty and 2) by using background music. Results showed that the difficulty adaptation induced a higher flow state, however music had no effect. There was a positive correlation between subjective flow scores and offline performance, although the flow factors had no effect (adaptation) or negative effect (music) on online performance. Overall, favouring the flow state seems a promising approach for enhancing users' satisfaction, although its complexity requires more thorough investigations.

Jérémy Frey, Renaud Gervais, Thibault Lainé et al.

Teegi is an anthropomorphic and tangible avatar exposing a users' brain activity in real time. It is connected to a device sensing the brain by means of electroencephalog-raphy (EEG). Teegi moves its hands and feet and closes its eyes along with the person being monitored. It also displays on its scalp the associated EEG signals, thanks to a semi-spherical display made of LEDs. Attendees can interact directly with Teegi -- e.g. move its limbs -- to discover by themselves the underlying brain processes. Teegi can be used for scientific outreach to introduce neurotechnologies in general and brain-computer interfaces (BCI) in particular.

Jérémy Frey

Brain-computer interfaces (BCI) are promising communication devices between humans and machines. BCI based on non-invasive neuroimaging techniques such as electroencephalography (EEG) have many applications , however the dissemination of the technology is limited, in part because of the price of the hardware. In this paper we compare side by side two EEG amplifiers, the consumer grade OpenBCI and the medical grade g.tec g.USBamp. For this purpose, we employed an original montage, based on the simultaneous recording of the same set of electrodes. Two set of recordings were performed. During the first experiment a simple adapter with a direct connection between the amplifiers and the electrodes was used. Then, in a second experiment, we attempted to discard any possible interference that one amplifier could cause to the other by adding "ideal" diodes to the adapter. Both spectral and temporal features were tested -- the former with a workload monitoring task, the latter with an visual P300 speller task. Overall, the results suggest that the OpenBCI board -- or a similar solution based on the Texas Instrument ADS1299 chip -- could be an effective alternative to traditional EEG devices. Even though a medical grade equipment still outperforms the OpenBCI, the latter gives very close EEG readings, resulting in practice in a classification accuracy that may be suitable for popularizing BCI uses.

Jérémy Frey

Nowadays computer science can create digital worlds that deeply immerse users; it can also process in real time brain activity to infer their inner states. What marvels can we achieve with such technologies? Go back to displaying text. And unfold a story that follows and molds users as never before.

Renaud Gervais, Joan Sol Roo, Jérémy Frey et al.

Digital devices are now ubiquitous and have the potential to be used to support positive changes in human lives and promote psychological well-being. This paper presents three interactive systems that we created focusing on introspection activities, leveraging tangible interaction and spatial augmented reality. More specifically, we describe anthropomorphic augmented avatars that display the users' inner states using physiological sensors. We also present a first prototype of an augmented sandbox specifically dedicated to promoting mindfulness activities.

Jérémy Frey

While physiological sensors enter the mass market and reach the general public, they are still mainly employed to monitor health -- whether it is for medical purpose or sports. We describe an application that uses heart rate feedback as an incentive for social interactions. A traditional board game has been "augmented" through remote physiological sensing, using webcams. Projection helped to conceal the technological aspects from users. We detail how players reacted -- stressful situations could emerge when users are deprived from their own signals -- and we give directions for game designers to integrate physiological sensors.

Jérémy Frey, Maxime Daniel, Julien Castet et al.

Measuring brain activity with electroencephalography (EEG) is mature enough to assess mental states. Combined with existing methods, such tool can be used to strengthen the understanding of user experience. We contribute a set of methods to estimate continuously the user's mental workload, attention and recognition of interaction errors during different interaction tasks. We validate these measures on a controlled virtual environment and show how they can be used to compare different interaction techniques or devices, by comparing here a keyboard and a touch-based interface. Thanks to such a framework, EEG becomes a promising method to improve the overall usability of complex computer systems.

Renaud Gervais, Jérémy Frey, Alexis Gay et al.

We propose a toolkit for creating Tangible Out-of-Body Experiences: exposing the inner states of users using physiological signals such as heart rate or brain activity. Tobe can take the form of a tangible avatar displaying live physiological readings to reflect on ourselves and others. Such a toolkit could be used by researchers and designers to create a multitude of potential tangible applications, including (but not limited to) educational tools about Science Technologies Engineering and Mathematics (STEM) and cognitive science, medical applications or entertainment and social experiences with one or several users or Tobes involved. Through a co-design approach, we investigated how everyday people picture their physiology and we validated the acceptability of Tobe in a scientific museum. We also give a practical example where two users relax together, with insights on how Tobe helped them to synchronize their signals and share a moment.

Dennis Wobrock, Jérémy Frey, Delphine Graeff et al.

Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usa-bility qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.

Jérémy Frey, Aurélien Appriou, Fabien Lotte et al.

With stereoscopic displays, a depth sensation that is too strong could impede visual comfort and result in fatigue or pain. Electroencephalography (EEG) is a technology which records brain activity. We used it to develop a novel brain-computer interface that monitors users' states in order to reduce visual strain. We present the first proof-of-concept system that discriminates comfortable conditions from uncomfortable ones during stereoscopic vision using EEG. It reacts within 1s to depth variations, achieving 63% accuracy on average and 74% when 7 consecutive variations are measured. This study could lead to adaptive systems that automatically suit stereoscopic displays to users and viewing conditions.

Jérémy Frey, Renaud Gervais, Stéphanie Fleck et al.

We introduce Teegi, a Tangible ElectroEncephaloGraphy (EEG) Interface that enables novice users to get to know more about something as complex as brain signals, in an easy, en- gaging and informative way. To this end, we have designed a new system based on a unique combination of spatial aug- mented reality, tangible interaction and real-time neurotech- nologies. With Teegi, a user can visualize and analyze his or her own brain activity in real-time, on a tangible character that can be easily manipulated, and with which it is possible to interact. An exploration study has shown that interacting with Teegi seems to be easy, motivating, reliable and infor- mative. Overall, this suggests that Teegi is a promising and relevant training and mediation tool for the general public.

Jérémy Frey

Physiological sensors are gaining the attention of manufacturers and users. As denoted by devices such as smartwatches or the newly released Kinect 2 -- which can covertly measure heartbeats -- or by the popularity of smartphone apps that track heart rate during fitness activities. Soon, physiological monitoring could become widely accessible and transparent to users. We demonstrate how one could take advantage of this situation to increase users' engagement and enhance user experience in human-agent interaction. We created an experimental protocol involving embodied agents -- "virtual avatars". Those agents were displayed alongside a beating heart. We compared a condition in which this feedback was simply duplicating the heart rates of users to another condition in which it was set to an average heart rate. Results suggest a superior social presence of agents when they display feedback similar to users' internal state. This physiological "similarity-attraction" effect may lead, with little effort, to a better acceptance of agents and robots by the general public.

Jérémy Frey, Léonard Pommereau, Fabien Lotte et al.

The conflict between vergence (eye movement) and accommodation (crystalline lens deformation) occurs in every stereoscopic display. It could cause important stress outside the "zone of comfort", when stereoscopic effect is too strong. This conflict has already been studied using questionnaires, during viewing sessions of several minutes. The present pilot study describes an experimental protocol which compares two different comfort conditions using electroencephalography (EEG) over short viewing sequences. Analyses showed significant differences both in event-related potentials (ERP) and in frequency bands power. An uncomfortable stereoscopy correlates with a weaker negative component and a delayed positive component in ERP. It also induces a power decrease in the alpha band and increases in theta and beta bands. With fast responses to stimuli, EEG is likely to enable the conception of adaptive systems, which could tune the stereoscopic experience according to each viewer.

Jérémy Frey, Christian Mühl, Fabien Lotte et al.

Evaluating human-computer interaction is essential as a broadening population uses machines, sometimes in sensitive contexts. However, traditional evaluation methods may fail to combine real-time measures, an "objective" approach and data contextualization. In this review we look at how adding neuroimaging techniques can respond to such needs. We focus on electroencephalography (EEG), as it could be handled effectively during a dedicated evaluation phase. We identify workload, attention, vigilance, fatigue, error recognition, emotions, engagement, flow and immersion as being recognizable by EEG. We find that workload, attention and emotions assessments would benefit the most from EEG. Moreover, we advocate to study further error recognition through neuroimaging to enhance usability and increase user experience.