13.0HCApr 8, 2024
Evaluation of an LLM in Identifying Logical Fallacies: A Call for Rigor When Adopting LLMs in HCI ResearchGionnieve Lim, Simon T. Perrault
There is increasing interest in the adoption of LLMs in HCI research. However, LLMs may often be regarded as a panacea because of their powerful capabilities with an accompanying oversight on whether they are suitable for their intended tasks. We contend that LLMs should be adopted in a critical manner following rigorous evaluation. Accordingly, we present the evaluation of an LLM in identifying logical fallacies that will form part of a digital misinformation intervention. By comparing to a labeled dataset, we found that GPT-4 achieves an accuracy of 0.79, and for our intended use case that excludes invalid or unidentified instances, an accuracy of 0.90. This gives us the confidence to proceed with the application of the LLM while keeping in mind the areas where it still falls short. The paper describes our evaluation approach, results and reflections on the use of the LLM for our intended task.
1.2CYJan 22, 2022
From 996 to 007: Challenges of Working from Home During the Epidemic in ChinaJie Gao, Pin Sym Foong, Yifan Yang et al.
During the COVID-19 epidemic in China, millions of workers in tech companies had to start working from home (WFH). The change was sudden, unexpected and companies were not ready for it. Additionally, it was also the first time that WFH was experienced on such a large scale. We used the opportunity to describe the effect of WFH at scale for a sustained period of time. As the lockdown was easing, we conducted semi-structured interviews with 12 participants from China working in tech companies. While at first, WFH was reported as a pleasant experience with advantages, e.g. flexible schedule, more time with family, over time, this evolved into a rather negative experience where workers start working all day, every day and feel a higher workload despite the actual workload being reduced. We discuss these results and how they could apply for other extreme circumstances and to help improve WFH in general.
Effectiveness of Area-to-Value Legends and Grid Lines in Contiguous Area CartogramsKelvin L. T. Fung, Simon T. Perrault, Michael T. Gastner
A contiguous area cartogram is a geographic map in which the area of each region is proportional to numerical data (e.g., population size) while keeping neighboring regions connected. In this study, we investigated whether value-to-area legends (square symbols next to the values represented by the squares' areas) and grid lines aid map readers in making better area judgments. We conducted an experiment to determine the accuracy, speed, and confidence with which readers infer numerical data values for the mapped regions. We found that, when only informed about the total numerical value represented by the whole cartogram without any legend, the distribution of estimates for individual regions was centered near the true value with substantial spread. Legends with grid lines significantly reduced the spread but led to a tendency to underestimate the values. Comparing differences between regions or between cartograms revealed that legends and grid lines slowed the estimation without improving accuracy. However, participants were more likely to complete the tasks when legends and grid lines were present, particularly when the area units represented by these features could be interactively selected. We recommend considering the cartogram's use case and purpose before deciding whether to include grid lines or an interactive legend.
12.4HCNov 19, 2020
Task-Based Effectiveness of Interactive Contiguous Area CartogramsIan K. Duncan, Shi Tingsheng, Simon T. Perrault et al.
Cartograms are map-based data visualizations in which the area of each map region is proportional to an associated numeric data value (e.g., population or gross domestic product). A cartogram is called contiguous if it conforms to this area principle while also keeping neighboring regions connected. Because of their distorted appearance, contiguous cartograms have been criticized as difficult to read. Some authors have suggested that cartograms may be more legible if they are accompanied by interactive features (e.g., animations, linked brushing, or infotips). We conducted an experiment to evaluate this claim. Participants had to perform visual analysis tasks with interactive and noninteractive contiguous cartograms. The task types covered various aspects of cartogram readability, ranging from elementary lookup tasks to synoptic tasks (i.e., tasks in which participants had to summarize high-level differences between two cartograms). Elementary tasks were carried out equally well with and without interactivity. Synoptic tasks, by contrast, were more difficult without interactive features. With access to interactivity, however, most participants answered even synoptic questions correctly. In a subsequent survey, participants rated the interactive features as "easy to use" and "helpful." Our study suggests that interactivity has the potential to make contiguous cartograms accessible even for those readers who are unfamiliar with interactive computer graphics or do not have a prior affinity to working with maps. Among the interactive features, animations had the strongest positive effect, so we recommend them as a minimum of interactivity when contiguous cartograms are displayed on a computer screen.
7.9HCOct 15, 2020
Local Perceptions and Practices of News Sharing and Fake NewsGionnieve Lim, Simon T. Perrault
Fake news is a prevalent problem, particularly in digital media, that undermines trust and cooperation among people. As a variety of global mitigation efforts arise, the understanding of how people consider fake news becomes important, especially in local contexts. To that end, we carried out a survey with 75 participants in Singapore to understand people's perceptions of and practices with news (real and fake). Locally, fake news was found to be more pervasive in instant messaging apps than in social media, with the problem attributed more strongly to sharing than to creation. Good news sharing practices were generally observed. Highest trust was reported in government communication platforms across 11 media items. These results show that Singapore possesses a peculiar sociocultural scene, suggesting that efforts directed towards locally relevant measures may be more effective in addressing fake news in Singapore. We detail the survey results and recommended directions in this paper.
3.3HCJul 26, 2020
Towards Inclusive Design for Privacy and Security Perspectives from an Aging SocietyPavithren V S Pakianathan, Simon Perrault
Over the past few years, older adults in Singapore have been massively connecting to the Internet using Smartphone. However due to the ever-changing nature of Technology and Cybersecurity landscape, an older adult's limited technical and Privacy and Security (P & S) knowledge, experience and declining cognitive and physical abilities puts them at higher risks. Furthermore mainstream smartphone applications, which are generally not designed with older adults in mind, could result in mismatched mental models thereby creating usability issues. We interviewed 10 older adults above 65 and 10 adults assisting them based in Singapore to investigate how smartphone P & S can be redesigned inclusively by addressing the needs of older adults and people who support them. Our results show that socio-cultural factors affected the process of getting or providing P & S help, culture and attitude affected learning behaviours and older adults expressed heterogeneous P & S preferences based on contextual factors and level of convenience, however there are opportunities for the mechanisms to be senior-friendly. Due to the complex relationship between an older adult's milieu and technology, we aim to utilize a technology probe to investigate further and contribute towards an inclusive P & S model.
3.3HCMay 28, 2020
Investigating Performance and Usage of Input Methods for Soft Keyboard HotkeysKatherine Fennedy, Sylvain Malacria, Hyowon Lee et al.
Touch-based devices, despite their mainstream availability, do not support a unified and efficient command selection mechanism, available on every platform and application. We advocate that hotkeys, conventionally used as a shortcut mechanism on desktop computers, could be generalized as a command selection mechanism for touch-based devices, even for keyboard-less applications. In this paper, we investigate the performance and usage of soft keyboard shortcuts or hotkeys (abbreviated SoftCuts) through two studies comparing different input methods across sitting, standing and walking conditions. Our results suggest that SoftCuts not only are appreciated by participants but also support rapid command selection with different devices and hand configurations. We also did not find evidence that walking deters their performance when using the Once input method.
3.3HCMar 31, 2020
Vibrotactile Feedback for Vertical 2D Space ExplorationLancelot Dupont, Christophe Jouffrais, Simon T. Perrault
Visually impaired people encounter many challenges in their everyday life, especially when it comes to navigating and representing space. The issue of shopping is addressed mostly on the level of navigation and product detection, but conveying clues about the object position to the user is rarely implemented. This work presents a prototype of vibrotactile wristband using spatiotemporal patterns to help visually impaired users reach an object in the 2D plane in front of them. A pilot study on twelve blindfolded sighted subjects showed that discretizing space in a seven by seven targets matrix and conveying clues with a discrete pattern on the vertical axis and a continuous pattern on the horizontal axis is an intuitive and effective design.
3.3HCMar 20, 2020
It's All in the Timing: Principles of Transient Distraction Illustrated with Vibrotactile TasksChristopher L. Asplund, Takashi Obana, Parag Bhatnagar et al.
Vibration is an efficient way of conveying information from a device to its user, and it is increasingly used for wrist or finger-worn devices such as smart rings. Unexpected vibrations or sounds from the environment may disrupt the perception of such information. Although disruptive effects have been systematically explored in vision and audition, they have been less examined in the haptic domain. Here we briefly review the relevant literature from HCI and psychology, distilling principles of when distraction is likely. We then investigate these principles through four experiments, examining how the timing and modality of relatively rare or unexpected stimuli (surprise distractors) affects the detection and recognition of vibrotactile target patterns. At short distractor-target delays (< 350 ms), both auditory and vibrotactile surprise distractors impaired performance. At a longer delay (1050 ms), performance was not affected overall, even being improved with repeated exposure to the vibrotactile distractors. We discuss the importance of our findings in the context of HCI and cognitive psychology, and we provide design guidelines for mitigating the effects of distraction on haptic devices.
5.8HCJan 14, 2020
Nudge for Deliberativeness: How Interface Features Influence Online DiscourseSanju Menon, Weiyu Zhang, Simon T. Perrault
Cognitive load is a significant challenge to users for being deliberative. Interface design has been used to mitigate this cognitive state. This paper surveys literature on the anchoring effect, partitioning effect and point-of-choice effect, based on which we propose three interface nudges, namely, the word-count anchor, partitioning text fields, and reply choice prompt. We then conducted a 2*2*2 factorial experiment with 80 participants (10 for each condition), testing how these nudges affect deliberativeness. The results showed a significant positive impact of the word-count anchor. There was also a significant positive impact of the partitioning text fields on the word count of response. The reply choice prompt showed a surprisingly negative affect on the quantity of response, hinting at the possibility that the reply choice prompt induces a fear of evaluation, which could in turn dampen the willingness to reply.