Tommy Nilsson

Oliver Bensch, Leonie Bensch, Tommy Nilsson et al.

This paper describes the capabilities and potential of the intelligent personal assistant (IPA) CORE (Checklist Organizer for Research and Exploration), designed to support astronauts during procedures onboard the International Space Station (ISS), the Lunar Gateway station, and beyond. We reflect on the importance of a reliable and flexible assistant capable of offline operation and highlight the usefulness of audiovisual interaction using augmented reality elements to intuitively display checklist information. We argue that current approaches to the design of IPAs in space operations fall short of meeting these criteria. Therefore, we propose CORE as an assistant that combines Knowledge Graphs (KGs), Retrieval-Augmented Generation (RAG) for a Generative Pre-Trained Transformer (GPT), and Augmented Reality (AR) elements to ensure an intuitive understanding of procedure steps, reliability, offline availability, and flexibility in terms of response style and procedure updates.

Oliver Bensch, Leonie Bensch, Tommy Nilsson et al.

As humanity prepares for new missions to the Moon and Mars, astronauts will need to operate with greater autonomy, given the communication delays that make real-time support from Earth difficult. For instance, messages between Mars and Earth can take up to 24 minutes, making quick responses impossible. This limitation poses a challenge for astronauts who must rely on in-situ tools to access the large volume of data from spacecraft sensors, rovers, and satellites, data that is often fragmented and difficult to use. To bridge this gap, systems like the Mars Exploration Telemetry-Driven Information System (METIS) are being developed. METIS is an AI assistant designed to handle routine tasks, monitor spacecraft systems, and detect anomalies, all while reducing the reliance on mission control. Current Generative Pretrained Transformer (GPT) Models, while powerful, struggle in safety-critical environments. They can generate plausible but incorrect responses, a phenomenon known as "hallucination," which could endanger astronauts. To overcome these limitations, this paper proposes enhancing systems like METIS by integrating GPTs, Retrieval-Augmented Generation (RAG), Knowledge Graphs (KGs), and Augmented Reality (AR). The idea is to allow astronauts to interact with their data more intuitively, using natural language queries and visualizing real-time information through AR. KGs will be used to easily access live telemetry and multimodal data, ensuring that astronauts have the right information at the right time. By combining AI, KGs, and AR, this new system will empower astronauts to work more autonomously, safely, and efficiently during future space missions.

Tommy Nilsson, Joel E. Fischer, Andy Crabtree et al.

UbiComp has been envisioned to bring about a future dominated by calm computing technologies making our everyday lives ever more convenient. Yet the same vision has also attracted criticism for encouraging a solitary and passive lifestyle. The aim of this paper is to explore and elaborate these tensions further by examining the human values surrounding future domestic UbiComp solutions. Drawing on envisioning and contravisioning, we probe members of the public (N=28) through the presentation and focus group discussion of two contrasting animated video scenarios, where one is inspired by "calm" and the other by "engaging" visions of future UbiComp technology. By analysing the reasoning of our participants, we identify and elaborate a number of relevant values involved in balancing the two perspectives. In conclusion, we articulate practically applicable takeaways in the form of a set of key design questions and challenges.

Glyn Lawson, Emily Shaw, Tessa Roper et al.

Virtual environments (VEs) offer potential benefits to health and safety training: exposure to dangerous (virtual) environments; the opportunity for experiential learning; and a high level of control over the training, in that aspects can be repeated or reviewed based on the trainee's performance. However, VEs are typically presented as audiovisual (AV) systems, whereas engagement of other senses could increase the immersion in the virtual experience. Moreover, other senses play a key role in certain health and safety contexts, for example the feel of heat and smell in a fire or smell in a fuel leak. A multisensory (MS) VE was developed, which provided simulated heat and smell in accordance with events in a virtual world. As users approached a virtual fire, they felt heat from three 2 kW heaters and smelled smoke from a scent diffuser. Behaviours in the MS VE demonstrated higher validity than those in a comparable AV VE, which ratings and verbatim responses indicated was down to a greater belief that participants were in a real fire. However, a study of the effectiveness of the MS VE as a training tool demonstrated that it did not offer benefits over AV as measured by a written knowledge test and subjective ratings of engagement, attitude towards health and safety and desire to repeat. However, the study found further evidence for the use of AV VEs in health and safety training, particularly as the subjective ratings were generally better than for PowerPoint based training. Despite the lack of evidence for MS simulation on traditional measures of training, the different attitudes and experiences of users suggest that it may have value as a system for changing trainees' attitudes towards their personal safety and awareness. This view was supported by feedback from industrial partners.

Tommy Nilsson, Andy Crabtree, Joel Fischer et al.

The domestic environment is a key area for the design and deployment of autonomous systems. Yet research indicates their adoption is already being hampered by a variety of critical issues including trust, privacy and security. This paper explores how potential users relate to the concept of autonomous systems in the home and elaborates further points of friction. It makes two contributions. One methodological, focusing on the use of provocative utopian and dystopian scenarios of future autonomous systems in the home. These are used to drive an innovative workshop-based approach to breaching experiments, which surfaces the usually tacit and unspoken background expectancies implicated in the organisation of everyday life that have a powerful impact on the acceptability of future and emerging technologies. The other contribution is substantive, produced through participants efforts to repair the incongruity or "reality disjuncture" created by utopian and dystopian visions, and highlights the need to build social as well as computational accountability into autonomous systems, and to enable coordination and control.

Emily Shaw, Tessa Roper, Tommy Nilsson et al.

Understanding validity of user behaviour in Virtual Environments (VEs) is critical as they are increasingly being used for serious Health and Safety applications such as predicting human behaviour and training in hazardous situations. This paper presents a comparative study exploring user behaviour in VE-based fire evacuation and investigates whether this is affected by the addition of thermal and olfactory simulation. Participants (N=43) were exposed to a virtual fire in an office building. Quantitative and qualitative analyses of participant attitudes and behaviours found deviations from those we would expect in real life (e.g. pre-evacuation actions), but also valid behaviours like fire avoidance. Potentially important differences were found between multisensory and audiovisual-only conditions (e.g. perceived urgency). We conclude VEs have significant potential in safety-related applications, and that multimodality may afford additional uses in this context, but the identified limitations of behavioural validity must be carefully considered to avoid misapplication of the technology.

Kevin K. Bowden, Tommy Nilsson, Christine P. Spencer et al.

By utilizing different communication channels, such as verbal language, gestures or facial expressions, virtually embodied interactive humans hold a unique potential to bridge the gap between human-computer interaction and actual interhuman communication. The use of virtual humans is consequently becoming increasingly popular in a wide range of areas where such a natural communication might be beneficial, including entertainment, education, mental health research and beyond. Behind this development lies a series of technological advances in a multitude of disciplines, most notably natural language processing, computer vision, and speech synthesis. In this paper we discuss a Virtual Human Journalist, a project employing a number of novel solutions from these disciplines with the goal to demonstrate their viability by producing a humanoid conversational agent capable of naturally eliciting and reacting to information from a human user. A set of qualitative and quantitative evaluation sessions demonstrated the technical feasibility of the system whilst uncovering a number of deficits in its capacity to engage users in a way that would be perceived as natural and emotionally engaging. We argue that naturalness should not always be seen as a desirable goal and suggest that deliberately suppressing the naturalness of virtual human interactions, such as by altering its personality cues, might in some cases yield more desirable results.

Tommy Nilsson

The application of mobile computing is currently altering patterns of our behavior to a greater degree than perhaps any other invention. In combination with the introduction of BLE (Bluetooth Low Energy) and similar technologies enabling context-awareness, designers are today finding themselves empowered to build experiences and facilitate interactions with our physical surroundings in ways not possible before. The aim of this thesis is to present a research project, currently underway at the University of Cambridge, which is dealing with implementation of a BLE system into a museum environment. By assessing the technology, describing the design decisions as well as presenting a qualitative evaluation, this paper seeks to provide insight into some of the challenges and possible solutions connected to the process of developing ubiquitous BLE computing systems for public spaces. The project outcome revealed the potential use of BLE to engage whole new groups of audiences as well as made me argue in favor of a more seamful approach to the design of these systems.

Jakub Binter, Kateřina Klapilová, Tereza Zikánová et al.

There are many reasons for utilising 3D animation and virtual reality in sexuality research. Apart from providing a mean with which to (re)experience certain situations there are four main advantages: a) bespoke animated stimuli can be created and customized, which is especially important when researching paraphilia and sexual preferences, b) stimulus production is less expensive and easier to produce compared to real world stimuli, c) virtual reality allows us to capture data such as physiological reasons to stimuli, that we would not be able to otherwise (without resorting to self-report measures which are especially problematic in this research domain), d) ethical, legal, and health and safety issues are less complex since neither physical nor psychological harm is caused to animated characters allowing for the safe presentation of stimuli involving vulnerable targets. The animation sub-group has been exploring so far several production quality levels and various animation procedures in a number of available software. The aim is to develop static as well as dynamic, interactive sexual stimuli for sexual diagnostic and therapeutic purposes. We are aware of number of ethical issues related to the use of virtual reality in proposed research are analysed in this chapter.

Tommy Nilsson, Alan Blackwell, Carl Hogsden et al.

BLE (Bluetooth Low Energy) is a new wireless communication technology that, thanks to reduced power consumption, promises to facilitate communication between computing devices and help us harness their power in environments and contexts previously untouched by information technology. Museums and other facilities housing various cultural content are a particularly interesting area of application. The University of Cambridge Museums consortium has put considerable effort into researching the potential uses of emerging technologies such as BLE to unlock new experiences enriching the way we engage with cultural information. As a part of this research initiative, our ambition has been to examine the challenges and opportunities introduced by the introduction of a BLE-centred system into the museum context. We present an assessment of the potential offered by this technology and of the design approaches that might yield the best results when developing BLE-centred experiences for museum environments. A pivotal part of our project consisted of designing, developing and evaluating a prototype mobile location-based BLE-centred game. A number of technical problems, such as unstable and fluctuating signal strength, were encountered throughout the project lifecycle. Instead of attempting to eliminate such problems, we argued in favour of embracing them and turning them into a cornerstone of the gameplay. Our study suggested that this alternative seamful design approach yields particularly good results when deploying the technology in public environments. The project outcome also demonstrated the potential of BLE-centred solutions to reach out and engage new demographics, especially children, extending their interest in museum visits.

Tommy Nilsson, Carl Hogsden, Charith Perera et al.

The application of mobile computing is currently altering patterns of our behavior to a greater degree than perhaps any other invention. In combination with the introduction of power efficient wireless communication technologies, such as Bluetooth Low Energy (BLE), designers are today increasingly empowered to shape the way we interact with our physical surroundings and thus build entirely new experiences. However, our evaluations of BLE and its abilities to facilitate mobile location-based experiences in public environments revealed a number of potential problems. Most notably, the position and orientation of the user in combination with various environmental factors, such as crowds of people traversing the space, were found to cause major fluctuations of the received BLE signal strength. These issues are rendering a seamless functioning of any location-based application practically impossible. Instead of achieving seamlessness by eliminating these technical issues, we thus choose to advocate the use of a seamful approach, i.e. to reveal and exploit these problems and turn them into a part of the actual experience. In order to demonstrate the viability of this approach, we designed, implemented and evaluated the Ghost Detector - an educational location-based museum game for children. By presenting a qualitative evaluation of this game and by motivating our design decisions, this paper provides insight into some of the challenges and possible solutions connected to the process of developing location-based BLE-enabled experiences for public cultural spaces.