Anders Sundnes Løvlie

Matthias Freiberger, Peter Kun, Christian Igel et al.

Models leveraging both visual and textual data such as Contrastive Language-Image Pre-training (CLIP), are the backbone of many recent advances in artificial intelligence. In this work, we show that despite their versatility, such models are vulnerable to what we refer to as fooling master images. Fooling master images are capable of maximizing the confidence score of a CLIP model for a significant number of widely varying prompts, while being either unrecognizable or unrelated to the attacked prompts for humans. The existence of such images is problematic as it could be used by bad actors to maliciously interfere with CLIP-trained image retrieval models in production with comparably small effort as a single image can attack many different prompts. We demonstrate how fooling master images for CLIP (CLIPMasterPrints) can be mined using stochastic gradient descent, projected gradient descent, or blackbox optimization. Contrary to many common adversarial attacks, the blackbox optimization approach allows us to mine CLIPMasterPrints even when the weights of the model are not accessible. We investigate the properties of the mined images, and find that images trained on a small number of image captions generalize to a much larger number of semantically related captions. We evaluate possible mitigation strategies, where we increase the robustness of the model and introduce an approach to automatically detect CLIPMasterPrints to sanitize the input of vulnerable models. Finally, we find that vulnerability to CLIPMasterPrints is related to a modality gap in contrastive pre-trained multi-modal networks. Code available at https://github.com/matfrei/CLIPMasterPrints.

Matthias Freiberger, Peter Kun, Anders Sundnes Løvlie et al.

Due to their architecture and how they are trained, artificial neural networks are typically not robust toward pruning or shuffling layers at test time. However, such properties would be desirable for different applications, such as distributed neural network architectures where the order of execution cannot be guaranteed or parts of the network can fail during inference. In this work, we address these issues through a number of training approaches for vision transformers whose most important component is randomizing the execution order of attention modules at training time. With our proposed approaches, vision transformers are capable to adapt to arbitrary layer execution orders at test time assuming one tolerates a reduction (about 20\%) in accuracy at the same model size. We analyse the feature representations of our trained models as well as how each layer contributes to the models prediction based on its position during inference. Our analysis shows that layers learn to contribute differently based on their position in the network. Finally, we layer-prune our models at test time and find that their performance declines gracefully. Code available at https://github.com/matfrei/layershuffle.

Louie Søs Meyer, Johanne Engel Aaen, Anitamalina Regitse Tranberg et al.

This Research through Design paper explores how object detection may be applied to a large digital art museum collection to facilitate new ways of encountering and experiencing art. We present the design and evaluation of an interactive application called SMKExplore, which allows users to explore a museum's digital collection of paintings by browsing through objects detected in the images, as a novel form of open-ended exploration. We provide three contributions. First, we show how an object detection pipeline can be integrated into a design process for visual exploration. Second, we present the design and development of an app that enables exploration of an art museum's collection. Third, we offer reflections on future possibilities for museums and HCI researchers to incorporate object detection techniques into the digitalization of museums.

Anders Sundnes Løvlie, Astrid Waagstein, Peter Hyldgård

This article reports on a Research through Design study exploring how to design a tool for helping readers of science journalism understand the strength and uncertainty of scientific evidence in news stories about health science, using both textual and visual information. A central aim has been to teach readers about criteria for assessing scientific evidence, in particular in order to help readers differentiate between science and pseudoscience. Working in a research-in-the-wild collaboration with a website for popular science, the study presents the design and evaluation of the Scientific Evidence Indicator, which uses metadata about scientific publications to present an assessment of evidence strength to the readers. Evaluations of the design demonstrate some success in helping readers recognize whether studies have undergone scientific peer review or not, but point to challenges in facilitating a more in-depth understanding. Insights from the study point to a potential for developing similar tools aimed at journalists rather than directly at audiences.

Anders Sundnes Løvlie, Astrid Waagstein, Peter Hyldgård

To what extent can news media help in providing more credible information about science? This is the core challenge for the Science Evidence Indicator (SEI) project, a collaboration between the Danish popular news website videnskab.dk and the authors of this paper. Looking specifically at medical science news, we aim to provide a transparent assessment of the scientific sources behind a story. This entails identifying some of the criteria that scientists use to assess research, and making it accessible and understandable for readers. We address the following research question: How can we communicate the quality of scientific publications in health science to a non-expert audience? Our goal is to make the assessments understandable for the youngest part of the website's target audience: high school students from age 16 and upwards.

David Kadish, Sebastian Risi, Anders Sundnes Løvlie

Despite recent advances in object detection using deep learning neural networks, these neural networks still struggle to identify objects in art images such as paintings and drawings. This challenge is known as the cross depiction problem and it stems in part from the tendency of neural networks to prioritize identification of an object's texture over its shape. In this paper we propose and evaluate a process for training neural networks to localize objects - specifically people - in art images. We generate a large dataset for training and validation by modifying the images in the COCO dataset using AdaIn style transfer. This dataset is used to fine-tune a Faster R-CNN object detection network, which is then tested on the existing People-Art testing dataset. The result is a significant improvement on the state of the art and a new way forward for creating datasets to train neural networks to process art images.

Anders Sundnes Løvlie, Karin Ryding, Jocelyn Spence et al.

This article brings together two distinct, but related perspectives on playful museum experiences: Critical play and hybrid design. The article explores the challenges involved in combining these two perspectives, through the design of two hybrid museum experiences that aimed to facilitate critical play with/in the collections of the Museum of Yugoslavia and the highly contested heritage they represent. Based on reflections from the design process as well as feedback from test users we describe a series of challenges: Challenging the norms of visitor behaviour, challenging the role of the artefact, and challenging the curatorial authority. In conclusion we outline some possible design strategies to address these challenges.

Petros Ioannidis, Lina Eklund, Anders Sundnes Løvlie

In this article, we present a lifecycle study of We Dare You, a Substitutional Reality (SR) installation that combines visual and tactile stimuli. The installation is set up in a center for architecture, and invites visitors to explore its facade while playing with vertigo, in a visual Virtual Reality (VR) environment that replicates the surrounding physical space of the installation. Drawing on an ethnographic approach, including observations and interviews, we researched the exhibit from its opening, through the initial months plagued by technical problems, its subsequent success as a social and playful installation, on to its closure, due to COVID-19, and its subsequent reopening. Our findings explore the challenges caused by both the hybrid nature of the installation, as well as the visitor' playful use of the installation which made the experience social and performative - but also caused some problems. We also discuss the problems We Dare You faced in light of hygiene demands due to COVID-19. The analysis contrasts the design processes and expectations of stakeholders with the audience's playful appropriation, which led the stakeholders to see the installation as both a success and a failure. Evaluating the design and redesign through use on behalf of visitors, we argue that an approach that further opens up the post-production experience to a process of continuous redesign based on the user input - what has been termed "design-after-design" - could facilitate the design of similar experiences in the museum and heritage sector, supporting a participatory agenda in the design process, and helping to resolve the tension between stakeholders' expectations and visitors' playful appropriations.

Anders Sundnes Løvlie, Lina Eklund, Annika Waern et al.

What does the age of participation look like from the perspective of a museum visitor? Arguably, the concept of participative experiences is already so deeply ingrained in our culture that we may not even think about it as participation. Museum visitors engage in a number of activities, of which observing the exhibits is only one part. Since most visitors come to the museum together with someone else, they spend time and attention on the people they came with, and often the needs of the group are given priority over individual preferences. How can museums tap into these activities - and make themselves relevant to visitors? In this chapter we will try to approach this constructively, as a design opportunity. Could it be productive for the museum to consider itself not only as a disseminator of knowledge, but also as the facilitator of participative activities between visitors? In what follows, we will outline a range of practical design projects that serve as examples of this approach. These projects were part of the European Union funded Horizon2020 project GIFT, a cross-disciplinary collaboration between researchers, artists, designers and many international museums and heritage organisations, exploring the concept of interpersonal museum experiences (see https://gifting.digital/). What the projects have in common is that they build on visitors co-creating and sharing their own narratives in the museum context. We suggest that these projects demonstrate a spectrum of possibilities: From experiences that take place almost without any museum involvement, to those that give museums a role in curating these narratives.

Karin Ryding, Jocelyn Spence, Anders Sundnes Løvlie et al.

We reflect on two museum visiting experiences that adopted the strategy of interpersonalization in which one visitor creates an experience for another. In the Gift app, visitors create personal mini-tours for specific others. In Never let me go, one visitor controls the experience of another by sending them remote instructions as they follow them around the museum. By reflecting on the design of these experiences and their deployment in museums we show how interpersonalization can deliver engaging social visits in which visitors make their own interpretations. We contrast the approach to previous research in customization and algorithmic personalization. We reveal how these experiences relied on intimacy between pairs of visitors but also between visitors and the museum. We propose that interpersonalization requires museums to step back to make space for interpretation, but that this then raises the challenge of how to reintroduce the museum's own perspective. Finally, we articulate strategies and challenges for applying this approach.

Mette Muxoll Schou, Anders Sundnes Løvlie

This paper presents a research through design exploration using tangible interactions in order to seamlessly integrate technology in a historical house museum. The study addresses a longstanding concern in museum exhibition design that interactive technologies may distract from the artifacts on display. Through an iterative design process including user studies, a co-creation workshop with museum staff and several prototypes, we developed an interactive installation called The Diary of Niels that combines physical objects, RFID sensors and an elaborate fiction in order to facilitate increased visitor engagement. Insights from the research process and user tests indicate that the integration of technology and artifacts is meaningful and engaging for users, and helps introduce museum visitors to the historic theme of the exhibition and the meaning of the artifacts. The study also points to continued challenges in integrating such hybrid experiences fully with the rest of the exhibition.