Kentaro Toyama

Jiyoon Kim, Kentaro Toyama, Sangmi Kim et al.

Generative AI challenges academic integrity not only by enabling students to delegate substantial portions of their academic work, but also by blurring the ethical boundaries by which students distinguish acceptable assistance from misconduct. Drawing on semi-structured interviews (n=20), AI chat logs, and course documents (syllabi, submitted assignments), we investigated how students themselves make moral sense of AI use in academic writing. Our analysis results in a range of novel findings: First, there are at least five distinct sites of AI-use conceptualization, ranging from faculty's intended AI policy, to students' actual AI use. Second, students use over 20 distinct rationalizations to justify AI use, such as that copying AI-generated text is victimless; that any AI text reflecting their own beliefs or their own style is their own writing; or that they are learning more by using AI -- even extensively -- than otherwise. We present a taxonomy of these rationalizations, and show how some of them are employed to justify conscious violations of course policies. Third, student rationalizations occur in both an ad hoc and post hoc manner, and they are not necessarily self-consistent. These and other findings suggest that modern AI presents a steep, ethical, slippery slope which students conceptually slide down, landing far outside the pedagogical goals and expectations of instructors. We discuss implications for educational design and AI policy.

Shan Randhawa, Agha Ali Raza, Kentaro Toyama et al.

LLMs are increasingly being integrated into clinical workflows, yet they often lack clinical empathy, an essential aspect of effective doctor-patient communication. Existing NLP frameworks focus on reactively labeling empathy in doctors' responses but offer limited support for anticipatory modeling of empathy needs, especially in general health queries. We introduce the Empathy Applicability Framework (EAF), a theory-driven approach that classifies patient queries in terms of the applicability of emotional reactions and interpretations, based on clinical, contextual, and linguistic cues. We release a benchmark of real patient queries, dual-annotated by Humans and GPT-4o. In the subset with human consensus, we also observe substantial human-GPT alignment. To validate EAF, we train classifiers on human-labeled and GPT-only annotations to predict empathy applicability, achieving strong performance and outperforming the heuristic and zero-shot LLM baselines. Error analysis highlights persistent challenges: implicit distress, clinical-severity ambiguity, and contextual hardship, underscoring the need for multi-annotator modeling, clinician-in-the-loop calibration, and culturally diverse annotation. EAF provides a framework for identifying empathy needs before response generation, establishes a benchmark for anticipatory empathy modeling, and enables supporting empathetic communication in asynchronous healthcare.

Jane Im, Kentaro Toyama

When an individual is harmed by someone in power, such as a workplace manager, it can help to identify allies--people who would offer sympathy, advice, or supportive action. However, ally discovery is fraught because the very people who might be most relevant--e.g., someone who reports to the same manager--might not be sympathetic and could potentially exacerbate the harm. We examine this problem in the specific context of PhD students navigating advising challenges and present a social media platform called "Moa" that brings together a number of features that we believe facilitate ally discovery. Moa's most novel element is an audience selection process that uses what we call consent boundaries, which allow users to flexibly define each post or comment's audience based on factors such as common social identity or lived experience, all while preserving anonymity--neither senders nor recipients learn each other's identities, even as the post reaches the right audience. A 3-week field study with 47 real-world users showed that the features in combination facilitated sensitive conversations about advising, with 22.6% of users using consent boundaries. We discuss both our overall "recipe" for systems for ally discovery and the benefits of a consent-centered approach to design.

Agrima Seth, Monojit Choudhary, Sunayana Sitaram et al.

Representational bias in large language models (LLMs) has predominantly been measured through single-response interactions and has focused on Global North-centric identities like race and gender. We expand on that research by conducting a systematic audit of GPT-4 Turbo to reveal how deeply encoded representational biases are and how they extend to less-explored dimensions of identity. We prompt GPT-4 Turbo to generate over 7,200 stories about significant life events (such as weddings) in India, using prompts designed to encourage diversity to varying extents. Comparing the diversity of religious and caste representation in the outputs against the actual population distribution in India as recorded in census data, we quantify the presence and "stickiness" of representational bias in the LLM for religion and caste. We find that GPT-4 responses consistently overrepresent culturally dominant groups far beyond their statistical representation, despite prompts intended to encourage representational diversity. Our findings also suggest that representational bias in LLMs has a winner-take-all quality that is more biased than the likely distribution bias in their training data, and repeated prompt-based nudges have limited and inconsistent efficacy in dislodging these biases. These results suggest that diversifying training data alone may not be sufficient to correct LLM bias, highlighting the need for more fundamental changes in model development. Dataset and Codebook: https://github.com/agrimaseth/How-Deep-Is-Representational-Bias-in-LLMs