Michael J. Lyons

Michael J. Lyons

Twenty-five years ago, my colleagues Miyuki Kamachi and Jiro Gyoba and I designed and photographed JAFFE, a set of facial expression images intended for use in a study of face perception. In 2019, without seeking permission or informing us, Kate Crawford and Trevor Paglen exhibited JAFFE in two widely publicized art shows. In addition, they published a nonfactual account of the images in the essay "Excavating AI: The Politics of Images in Machine Learning Training Sets." The present article recounts the creation of the JAFFE dataset and unravels each of Crawford and Paglen's fallacious statements. I also discuss JAFFE more broadly in connection with research on facial expression, affective computing, and human-computer interaction.

Michael J. Lyons, Daniel Kluender

We present three patterns for computer-mediated interaction which we discovered during the design and development of a platform for remote teaching and learning of kanji, the Chinese characters used in written Japanese. Our aim in developing this system was to provide a basis for embodiment in remote interaction, and in particular to support the experience of empathy by both teacher and student. From this study, the essential elements are abstracted and suggested as design patterns for other computer-mediated interaction systems.

Ivan Poupyrev, Michael J. Lyons, Sidney Fels et al.

The rapid evolution of electronics, digital media, advanced materials, and other areas of technology, is opening up unprecedented opportunities for musical interface inventors and designers. The possibilities afforded by these new technologies carry with them the challenges of a complex and often confusing array of choices for musical composers and performers. New musical technologies are at least partly responsible for the current explosion of new musical forms, some of which are controversial and challenge traditional definitions of music. Alternative musical controllers, currently the leading edge of the ongoing dialogue between technology and musical culture, involve many of the issues covered at past CHI meetings. This workshop brings together interface experts interested in musical controllers and musicians and composers involved in the development of new musical interfaces.

Tomohiro Tokunaga, Michael J. Lyons

We are exploring the design and implementation of artificial expressions, kinetic audio-visual representations of real-time physiological data that reflect emotional and cognitive state. In this work, we demonstrate a prototype, the Enactive Mandala, which maps real-time EEG signals to modulate ambient music and animated visual music. Transparent real-time audio-visual feedback of brainwave qualities supports intuitive insight into the connection between thoughts and physiological states.

Alexander Refsum Jensenius, Michael J. Lyons

This paper provides an overview of the process of editing the forthcoming anthology "A NIME Reader - Fifteen Years of New Interfaces for Musical Expression." The selection process is presented, and we reflect on some of the trends we have observed in re-discovering the collection of more than 1200 NIME papers published throughout the 15-year long history of the conference. An anthology is necessarily selective, and ours is no exception. As we present in this paper, the aim has been to represent the wide range of artistic, scientific, and technological approaches that characterize the NIME conference. The anthology also includes critical discourse, and through acknowledgment of the strengths and weaknesses of the NIME community, we propose activities that could further diversify and strengthen the field.

Gamhewage C. de Silva, Tamara Smyth, Michael J. Lyons

We describe a simple, computationally light, real-time system for tracking the lower face and extracting information about the shape of the open mouth from a video sequence. The system allows unencumbered control of audio synthesis modules by the action of the mouth. We report work in progress to use the mouth controller to interact with a physical model of sound production by the avian syrinx.

Mathias Funk, Kazuhiro Kuwabara, Michael J. Lyons

The central role of the face in social interaction and non-verbal communication suggests we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.

Michael J. Lyons, Michael Haehnel, Nobuji Tetsutani

The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.

Laurent Prevost, Olivier Liechti, Michael J. Lyons

This paper describes the design, implementation, and initial testing of a reusable platform for the creation of pervasive games with geo-localization services. We concentrate on role-playing games built by combining several types of simpler mini-games having three major components: Quests; Collectables; and Non-player characters (NPC). Quests encourage players to be active in their physical environment and take part in collaborative play; Collectables provide motivation; and NPCs enable player-friendly interaction with the platform. Each of these elements poses different technical requirements, which were met by implementing the gaming platform using the inTrack pervasive middle-ware being developed by our group. Several sample games were implemented and tested within the urban environment of Kyoto, Japan, using gaming clients running on mobile phones from NTT DoCoMo, Japan's largest mobile provider.

Michael J. Lyons

Considerable effort has been devoted to the automatic extraction of information about action of the face from image sequences. Within the context of human-computer interaction (HCI) we may distinguish systems that allow expression from those which aim at recognition. Most of the work in facial action processing has been directed at automatically recognizing affect from facial actions. By contrast, facial gesture interfaces, which respond to deliberate facial actions, have received comparatively little attention. This paper reviews several projects on vision-based interfaces that rely on facial action for intentional HCI. Applications to several domains are introduced, including text entry, artistic and musical expression and assistive technology for motor-impaired users.

Michael J. Lyons, Miyuki Kamachi, Jiro Gyoba

We present a method for extracting information about facial expressions from digital images. The method codes facial expression images using a multi-orientation, multi-resolution set of Gabor filters that are topographically ordered and approximately aligned with the face. A similarity space derived from this code is compared with one derived from semantic ratings of the images by human observers. Interestingly the low-dimensional structure of the image-derived similarity space shares organizational features with the circumplex model of affect, suggesting a bridge between categorical and dimensional representations of facial expression. Our results also indicate that it would be possible to construct a facial expression classifier based on a topographically-linked multi-orientation, multi-resolution Gabor coding of the facial images at the input stage. The significant degree of psychological plausibility exhibited by the proposed code may also be useful in the design of human-computer interfaces.

Michael J. Lyons

Two art exhibitions, "Training Humans" and "Making Faces," and the accompanying essay "Excavating AI: The politics of images in machine learning training sets" by Kate Crawford and Trevor Paglen, are making substantial impact on discourse taking place in the social and mass media networks, and some scholarly circles. Critical scrutiny reveals, however, a self-contradictory stance regarding informed consent for the use of facial images, as well as serious flaws in their critique of ML training sets. Our analysis underlines the non-negotiability of informed consent when using human data in artistic and other contexts, and clarifies issues relating to the description of ML training sets.

Michael J. Lyons

Commentary on the article Fourteen Years of NIME: The Value and Meaning of Community in Interactive Music Research by A. Marquez-Borbon and P. Stapleton.

Andreas Wiratanaya, Michael J. Lyons

We describe the design and implementation of a vision based interactive entertainment system that makes use of both involuntary and voluntary control paradigms. Unintentional input to the system from a potential viewer is used to drive attention-getting output and encourage the transition to voluntary interactive behaviour. The iMime system consists of a character animation engine based on the interaction metaphor of a mime performer that simulates non-verbal communication strategies, without spoken dialogue, to capture and hold the attention of a viewer. The system was developed in the context of a project studying care of dementia sufferers. Care for a dementia sufferer can place unreasonable demands on the time and attentional resources of their caregivers or family members. Our study contributes to the eventual development of a system aimed at providing relief to dementia caregivers, while at the same time serving as a source of pleasant interactive entertainment for viewers. The work reported here is also aimed at a more general study of the design of interactive entertainment systems involving a mixture of voluntary and involuntary control.

Michael J. Lyons

There is a perceived controversy as to whether the cognitive representation of affect is better modelled using a dimensional or categorical theory. This paper first suggests that these views are, in fact, compatible. The paper then discusses this theme and related issues in reference to a commonly stated application domain of research on human affect and expression: human computer interaction (HCI). The novel suggestion here is that a more realistic framing of studies of human affect in expression with reference to HCI and, particularly HCHI (Human-Computer-Human Interaction) entails some re-formulation of the approach to the basic phenomena themselves. This theme is illustrated with several examples from several recent research projects.

Chamin Morikawa, Michael J. Lyons

Interaction methods based on computer-vision hold the potential to become the next powerful technology to support breakthroughs in the field of human-computer interaction. Non-invasive vision-based techniques permit unconventional interaction methods to be considered, including use of movements of the face and head for intentional gestural control of computer systems. Facial gesture interfaces open new possibilities for assistive input technologies. This chapter gives an overview of research aimed at developing vision-based head and face-tracking interfaces. This work has important implications for future assistive input devices. To illustrate this concretely we describe work from our own research in which we developed two vision-based facial feature tracking algorithms for human computer interaction and assistive input. Evaluation forms a critical component of this research and we provide examples of new quantitative evaluation tasks as well as the use of model real-world applications for the qualitative evaluation of new interaction styles.

Michael J. Lyons

Position: (1) Partial solutions to machine intelligence can lead to systems which may be useful creating interesting and expressive musical works. (2) An appropriate general goal for this field is augmenting human expression. (3) The study of the aesthetics of human augmentation in musical performance is in its infancy.

Christoph Bartneck, Michael J. Lyons, Martin Saerbeck

Emotions play a central role in most forms of natural human interaction so we may expect that computational methods for the processing and expression of emotions will play a growing role in human-computer interaction. The OCC model has established itself as the standard model for emotion synthesis. A large number of studies employed the OCC model to generate emotions for their embodied characters. Many developers of such characters believe that the OCC model will be all they ever need to equip their character with emotions. This study reflects on the limitations of the OCC model specifically, and on the emotion models in general due to their dependency on artificial intelligence.