John Zimmerman

Oscar J. Romero, John Zimmerman, Aaron Steinfeld et al.

This paper explores the integration of two AI subdisciplines employed in the development of artificial agents that exhibit intelligent behavior: Large Language Models (LLMs) and Cognitive Architectures (CAs). We present three integration approaches, each grounded in theoretical models and supported by preliminary empirical evidence. The modular approach, which introduces four models with varying degrees of integration, makes use of chain-of-thought prompting, and draws inspiration from augmented LLMs, the Common Model of Cognition, and the simulation theory of cognition. The agency approach, motivated by the Society of Mind theory and the LIDA cognitive architecture, proposes the formation of agent collections that interact at micro and macro cognitive levels, driven by either LLMs or symbolic components. The neuro-symbolic approach, which takes inspiration from the CLARION cognitive architecture, proposes a model where bottom-up learning extracts symbolic representations from an LLM layer and top-down guidance utilizes symbolic representations to direct prompt engineering in the LLM layer. These approaches aim to harness the strengths of both LLMs and CAs, while mitigating their weaknesses, thereby advancing the development of more robust AI systems. We discuss the tradeoffs and challenges associated with each approach.

Shixian Xie, Motahhare Eslami, John Zimmerman

Despite significant advances in responsible AI research, industry adoption remains limited, leaving many HCI contributions underutilized in practice. This position paper argues that current research often fails to account for the fundamental need for capitalist enterprises to create value. To achieve immediate real-world impact, responsible AI research must explore how to design responsibly within capitalism. We call for a move beyond the dichotomy of "ethics vs. business" toward a more productive framing of "ethics and business." We propose ideation as a practical design strategy for generating ethically preferable alternatives that also meet business objectives. By aligning ethics with enterprise realities, we expand the space of responsible design that can actually be built.

Devansh Saxena, Ji-Youn Jung, Jodi Forlizzi et al.

AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance falls short of what is needed to ensure safety and co-create value. These mismatches are particularly difficult to address once development is underway, highlighting the need for early-stage intervention. Navigating complex, multi-dimensional risk factors that contribute to AI Mismatches is a persistent challenge. To address it, we propose an AI Mismatch approach to anticipate and mitigate risks early on, focusing on the gap between realistic model performance and required task performance. Through an analysis of 774 AI cases, we extracted a set of critical factors, which informed the development of seven matrices that map the relationships between these factors and highlight high-risk areas. Through case studies, we demonstrate how our approach can help reduce risks in AI development.

Minjung Park, Jodi Forlizzi, John Zimmerman

Innovators transform the world by understanding where services are successfully meeting customers' needs and then using this knowledge to identify failsafe opportunities for innovation. Pre-trained models have changed the AI innovation landscape, making it faster and easier to create new AI products and services. Understanding where pre-trained models are successful is critical for supporting AI innovation. Unfortunately, the hype cycle surrounding pre-trained models makes it hard to know where AI can really be successful. To address this, we investigated pre-trained model applications developed by HCI researchers as a proxy for commercially successful applications. The research applications demonstrate technical capabilities, address real user needs, and avoid ethical challenges. Using an artifact analysis approach, we categorized capabilities, opportunity domains, data types, and emerging interaction design patterns, uncovering some of the opportunity space for innovation with pre-trained models.

Alicia, Lee, Mai Lee Chang et al.

The increasing number of older adults who experience cognitive decline places a burden on informal caregivers, whose support with tasks of daily living determines whether older adults can remain in their homes. To explore how agents might help lower-SES older adults to age-in-place, we interviewed ten pairs of older adults experiencing cognitive decline and their informal caregivers. We explored how they coordinate care, manage burdens, and sustain autonomy and privacy. Older adults exercised control by delegating tasks to specific caregivers, keeping information about all the care they received from their adult children. Many abandoned some tasks of daily living, lowering their quality of life to ease caregiver burden. One effective strategy, piggybacking, uses spontaneous overlaps in errands to get more work done with less caregiver effort. This raises the questions: (i) Can agents help with piggyback coordination? (ii) Would it keep older adults in their homes longer, while not increasing caregiver burden?

Michal Luria, John Zimmerman, Jodi Forlizzi

One of the challenges in conducting research on the intersection of the CHI and Human-Robot Interaction (HRI) communities is in addressing the gap of acceptable design research methods between the two. While HRI is focused on interaction with robots and includes design research in its scope, the community is not as accustomed to exploratory design methods as the CHI community. This workshop paper argues for bringing exploratory design, and specifically Research through Design (RtD) methods that have been established in CHI for the past decade to the foreground of HRI. RtD can enable design researchers in the field of HRI to conduct exploratory design work that asks what is the right thing to design and share it within the community.

Qian Yang, Aaron Steinfeld, John Zimmerman

Clinical decision support tools (DST) promise improved healthcare outcomes by offering data-driven insights. While effective in lab settings, almost all DSTs have failed in practice. Empirical research diagnosed poor contextual fit as the cause. This paper describes the design and field evaluation of a radically new form of DST. It automatically generates slides for clinicians' decision meetings with subtly embedded machine prognostics. This design took inspiration from the notion of "Unremarkable Computing", that by augmenting the users' routines technology/AI can have significant importance for the users yet remain unobtrusive. Our field evaluation suggests clinicians are more likely to encounter and embrace such a DST. Drawing on their responses, we discuss the importance and intricacies of finding the right level of unremarkableness in DST design, and share lessons learned in prototyping critical AI systems as a situated experience.

Aaron Steinfeld, John Zimmerman, Anthony Tomasic

There is great promise in creating effective technology experiences during situationally-induced impairments and disabilities through the combination of universal design and adaptive interfaces. We believe this combination is a powerful approach for meeting the UX needs of people with disabilities, including those which are temporary in nature. Research in each of these areas, and the combination, illustrates this promise.

Ben Zamanzadeh, Naveen Ashish, Cartic Ramakrishnan et al.

We present the concept of Semantic Advertising which we see as the future of online advertising. Semantic Advertising is online advertising powered by semantic technology which essentially enables us to represent and reason with concepts and the meaning of things. This paper aims to 1) Define semantic advertising, 2) Place it in the context of broader and more widely used concepts such as the Semantic Web and Semantic Search, 3) Provide a survey of work in related areas such as context matching, and 4) Provide a perspective on successful emerging technologies and areas of future work. We base our work on our experience as a company developing semantic technologies aimed at realizing the full potential of online advertising.