Saiph Savage

Claudia Flores-Saviaga, Shangbin Feng, Saiph Savage

The limited information (data voids) on political topics relevant to underrepresented communities has facilitated the spread of disinformation. Independent journalists who combat disinformation in underrepresented communities have reported feeling overwhelmed because they lack the tools necessary to make sense of the information they monitor and address the data voids. In this paper, we present a system to identify and address political data voids within underrepresented communities. Armed with an interview study, indicating that the independent news media has the potential to address them, we designed an intelligent collaborative system, called Datavoidant. Datavoidant uses state-of-the-art machine learning models and introduces a novel design space to provide independent journalists with a collective understanding of data voids to facilitate generating content to cover the voids. We performed a user interface evaluation with independent news media journalists (N=22). These journalists reported that Datavoidant's features allowed them to more rapidly while easily having a sense of what was taking place in the information ecosystem to address the data voids. They also reported feeling more confident about the content they created and the unique perspectives they had proposed to cover the voids. We conclude by discussing how Datavoidant enables a new design space wherein individuals can collaborate to make sense of their information ecosystem and actively devise strategies to prevent disinformation.

Claudia Flores-Saviaga, Christopher Curtis, Saiph Savage

AI has revolutionized the processing of various services, including the automatic facial verification of people. Automated approaches have demonstrated their speed and efficiency in verifying a large volume of faces, but they can face challenges when processing content from certain communities, including communities of people of color. This challenge has prompted the adoption of "human-in-the-loop" (HITL) approaches, where human workers collaborate with the AI to minimize errors. However, most HITL approaches do not consider workers' individual characteristics and backgrounds. This paper proposes a new approach, called Inclusive Portraits (IP), that connects with social theories around race to design a racially-aware human-in-the-loop system. Our experiments have provided evidence that incorporating race into human-in-the-loop (HITL) systems for facial verification can significantly enhance performance, especially for services delivered to people of color. Our findings also highlight the importance of considering individual worker characteristics in the design of HITL systems, rather than treating workers as a homogenous group. Our research has significant design implications for developing AI-enhanced services that are more inclusive and equitable.

Kashif Imteyaz, Qiushi, Liang et al.

Blind and low-vision (BLV) individuals face high unemployment rates. The job search is becoming harder as more employers use AI-driven systems to screen resumes before a human ever sees them. Such AI systems could inadvertently further disadvantage BLV job seekers, introducing additional barriers to an already difficult process. We lack understanding of BLV job seekers' experiences in today's AI-driven hiring ecosystem. Without such understanding, we risk designing technologies that create new systemic barriers for BLV job seekers rather than providing support. To this end, we conducted interviews with 17 BLV job seekers and analyzed their experiences with AI-powered hiring systems. We found that AI hiring systems misrepresented their professional identities and created dehumanizing interactions. To level the playing field, BLV job seekers used strategic counter-navigation: they deployed their own tools to bypass algorithmic screening and built peer networks to share AI literacy. They also practiced 'strategic refusal', choosing to avoid certain AI systems to regain their agency. Unlike prior work that frames job search as an individualistic activity, or one focused on being compliant with employer needs, we use the interdependence framework to argue that for BLV people, job search is an interdependent process. We offer design recommendations for AI-mediated tools that center disability perspectives and support interdependencies in job search.

Christoph Riedl, Saiph Savage, Josie Zvelebilova

AI is not only a neutral tool in team settings; it actively reshapes the social and cognitive fabric of collaboration. We advance a unified framework of alignment in distributed cognition in human-AI teams -- a process through which linguistic, cognitive, and social coordination emerge as human and AI agents co-construct a shared representational space. Across two studies, we show that exposure to AI-generated language shapes not only how people speak, but also how they think, what they attend to, and how they relate to each other. Together, these findings reveal how AI participation reorganizes the distributed cognitive architecture of teams: AI systems function as implicit social forcefields. Our findings highlight the double-edged impact of AI: the same mechanisms that enable efficient collaboration can also erode epistemic diversity and undermine natural alignment processes. We argue for rethinking AI in teams as a socially influential actor and call for new design paradigms that foreground transparency, controllability, and group-level dynamics to foster responsible, productive human-AI collaboration.

Carlos Toxtli, Christopher Curtis, Saiph Savage

Crowdsourcing markets are expanding worldwide, but often feature standardized interfaces that ignore the cultural diversity of their workers, negatively impacting their well-being and productivity. To transform these workplace dynamics, this paper proposes creating culturally-aware workplace tools, specifically designed to adapt to the cultural dimensions of monochronic and polychronic work styles. We illustrate this approach with "CultureFit," a tool that we engineered based on extensive research in Chronemics and culture theories. To study and evaluate our tool in the real world, we conducted a field experiment with 55 workers from 24 different countries. Our field experiment revealed that CultureFit significantly improved the earnings of workers from cultural backgrounds often overlooked in design. Our study is among the pioneering efforts to examine culturally aware digital labor interventions. It also provides access to a dataset with over two million data points on culture and digital work, which can be leveraged for future research in this emerging field. The paper concludes by discussing the importance and future possibilities of incorporating cultural insights into the design of tools for digital labor.

Monica Munnangi, Saiph Savage

Medical question-answering benchmarks predominantly evaluate single-turn exchanges, failing to capture the iterative, clarification-seeking nature of real patient consultations. We introduce ThReadMed-QA, a benchmark of 2,437 fully-answered patient-physician conversation threads extracted from r/AskDocs, comprising 8,204 question-answer pairs across up to 9 turns. Unlike prior work relying on simulated dialogues, adversarial prompts, or exam-style questions, ThReadMed-QA captures authentic patient follow-up questions and verified physician responses, reflecting how patients naturally seek medical information online. We evaluate five state-of-the-art LLMs -- GPT-5, GPT-4o, Claude Haiku, Gemini 2.5 Flash, and Llama 3.3 70B -- on a stratified test split of 238 conversations (948 QA pairs) using a calibrated LLM-as-a-judge rubric grounded in physician ground truth. Even the strongest model, GPT-5, achieves only 41.2% fully-correct responses. All five models degrade significantly from turn 0 to turn 2 (p < 0.001), with wrong-answer rates roughly tripling by the third turn. We identify a fundamental tension between single-turn capability and multi-turn reliability: models with the strongest initial performance (GPT-5: 75.2; Claude Haiku: 72.3 out of 100) exhibit the steepest declines by turn 2 (dropping 16.2 and 25.0 points respectively), while weaker models plateau or marginally improve. We introduce two metrics to quantify multi-turn failure modes: Conversational Consistency Score (CCS) and Error Propagation Rate (EPR). CCS reveals that nearly one in three Claude Haiku conversations swings between a fully correct and a completely wrong response within the same thread. EPR shows that a single wrong turn raises the probability of a subsequent wrong turn by 1.9-6.1x across all models.

Kashif Imteyaz, Isabel Lopez, Nakul Rajpal et al.

Freelance workers must continually acquire new skills to remain competitive in online labor markets, yet they lack the organizational training, mentorship, and infrastructure available to traditional employees. Generative AI-powered tools like ChatGPT are reshaping market skill demands while also offering new forms of on-demand learning support to meet those demands. Despite growing interest in AI-powered learning tools, little is known about how freelancers actually use these tools to learn, the challenges they encounter, and how generative AI for learning interacts with precarity and competition in platform-based work. We present a mixed-methods study combining a survey and semi-structured interviews with freelance knowledge workers. Grounded in self-directed learning theory, we examine how freelancers integrate generative AI tools into their learning practices. Our findings show that freelancers increasingly rely on generative AI to structure learning and support exploratory skill acquisition, but do not treat it as their primary learning resource due to inconsistency, lack of contextual relevance, and verification overhead. We identify a shift from learning as growth to learning as survival, where upskilling is oriented toward immediate market viability rather than long-term development. We also surface a structural challenge we term invisible competencies, in which workers acquire skills through generative AI tools but lack credible ways to signal or validate these skills in competitive freelance markets. Based on these insights, we offer design recommendations for generative AI-powered learning tools for freelancers.

Claudia Flores-Saviaga, Benjamin V. Hanrahan, Kashif Imteyaz et al.

The rapid adoption of generative AI in software development has impacted the industry, yet its effects on developers with visual impairments remain largely unexplored. To address this gap, we used an Activity Theory framework to examine how developers with visual impairments interact with AI coding assistants. For this purpose, we conducted a study where developers who are visually impaired completed a series of programming tasks using a generative AI coding assistant. We uncovered that, while participants found the AI assistant beneficial and reported significant advantages, they also highlighted accessibility challenges. Specifically, the AI coding assistant often exacerbated existing accessibility barriers and introduced new challenges. For example, it overwhelmed users with an excessive number of suggestions, leading developers who are visually impaired to express a desire for ``AI timeouts.'' Additionally, the generative AI coding assistant made it more difficult for developers to switch contexts between the AI-generated content and their own code. Despite these challenges, participants were optimistic about the potential of AI coding assistants to transform the coding experience for developers with visual impairments. Our findings emphasize the need to apply activity-centered design principles to generative AI assistants, ensuring they better align with user behaviors and address specific accessibility needs. This approach can enable the assistants to provide more intuitive, inclusive, and effective experiences, while also contributing to the broader goal of enhancing accessibility in software development.

Gianna Williams, Maya De Los Santos, Alexandra To et al.

The global AI surge demands crowdworkers from diverse languages and cultures. They are pivotal in labeling data for enabling global AI systems. Despite global significance, research has primarily focused on understanding the perspectives and experiences of US and India crowdworkers, leaving a notable gap. To bridge this, we conducted a survey with 100 crowdworkers across 16 Latin American and Caribbean countries. We discovered that these workers exhibited pride and respect for their digital labor, with strong support and admiration from their families. Notably, crowd work was also seen as a stepping stone to financial and professional independence. Surprisingly, despite wanting more connection, these workers also felt isolated from peers and doubtful of others' labor quality. They resisted collaboration and gender-based tools, valuing gender-neutrality. Our work advances HCI understanding of Latin American and Caribbean crowdwork, offering insights for digital resistance tools for the region.

Carlos Toxtli, Siddharth Suri, Saiph Savage

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.

Claudia Flores-Saviaga, Yuwen Li, Benjamin V. Hanrahan et al.

Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd work-ers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural AmazonMechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of421 crowd workers from differing geographic regions in theU.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools

Claudia Flores-Saviaga, Jessica Hammer, Juan Pablo Flores et al.

Large-scale streaming platforms such as Twitch are becoming increasingly popular, but detailed audience-streamer interaction dynamics remain unexplored at scale. In this paper, we perform a mixed-methods study on a dataset with over 12 million audience chat messages and 45 hours of streaming video to understand audience participation and streamer performance on Twitch. We uncover five types of streams based on size and audience participation styles: Clique Streams, small streams with close streamer-audience interactions; Rising Streamers, mid-range streams using custom technology and moderators to formalize their communities; Chatter-boxes, mid-range streams with established conversational dynamics; Spotlight Streamers, large streams that engage large numbers of viewers while still retaining a sense of community; and Professionals, massive streams with the stadium-style audiences. We discuss challenges and opportunities emerging for streamers and audiences from each style and conclude by providing data-backed design implications that empower streamers, audiences, live streaming platforms, and game designers

Claudia Flores-Saviaga, Saiph Savage

Social media platforms have been extensively used during natural disasters. However, most prior work has lacked focus on studying their usage during disasters in the Global South, where Internet access and social media utilization differs from developing countries. In this paper, we study how social media was used in the aftermath of the 7.1-magnitude earthquake that hit Mexico on September 19 of 2017 (known as the #19S earthquake). We conduct an analysis of how participants utilized social media platforms in the #19S aftermath. Our research extends investigations of crisis informatics by: 1) examining how participants used different social media platforms in the aftermath of a natural disaster in a Global South country; 2) uncovering how individuals developed their own processes to verify news reports using an on-the-ground citizen approach; 3) revealing how people developed their own mechanisms to deal with outdated information. For this, we surveyed 356 people. Additionally, we analyze one month of activity from: Facebook (12,606 posts), Twitter (2,909,109 tweets), Slack (28,782 messages), and GitHub (2,602 commits). This work offers a multi-platform view on user behavior to coordinate relief efforts, reduce the spread of misinformation and deal with obsolete information which seems to have been essential to help in the coordination and efficiency of relief efforts. Finally, based on our findings, we make recommendations for technology design to improve the effectiveness of social media use during crisis response efforts and mitigate the spread of misinformation across social media platforms.

Carlos Toxtli, Angela Richmond-Fuller, Saiph Savage

Our study presents a new tool, Reputation Agent, to promote fairer reviews from requesters (employers or customers) on gig markets. Unfair reviews, created when requesters consider factors outside of a worker's control, are known to plague gig workers and can result in lost job opportunities and even termination from the marketplace. Our tool leverages machine learning to implement an intelligent interface that: (1) uses deep learning to automatically detect when an individual has included unfair factors into her review (factors outside the worker's control per the policies of the market); and (2) prompts the individual to reconsider her review if she has incorporated unfair factors. To study the effectiveness of Reputation Agent, we conducted a controlled experiment over different gig markets. Our experiment illustrates that across markets, Reputation Agent, in contrast with traditional approaches, motivates requesters to review gig workers' performance more fairly. We discuss how tools that bring more transparency to employers about the policies of a gig market can help build empathy thus resulting in reasoned discussions around potential injustices towards workers generated by these interfaces. Our vision is that with tools that promote truth and transparency we can bring fairer treatment to gig workers.

Saiph Savage, Chun-Wei Chiang, Susumu Saito et al.

Crowd markets have traditionally limited workers by not providing transparency information concerning which tasks pay fairly or which requesters are unreliable. Researchers believe that a key reason why crowd workers earn low wages is due to this lack of transparency. As a result, tools have been developed to provide more transparency within crowd markets to help workers. However, while most workers use these tools, they still earn less than minimum wage. We argue that the missing element is guidance on how to use transparency information. In this paper, we explore how novice workers can improve their earnings by following the transparency criteria of Super Turkers, i.e., crowd workers who earn higher salaries on Amazon Mechanical Turk (MTurk). We believe that Super Turkers have developed effective processes for using transparency information. Therefore, by having novices follow a Super Turker criteria (one that is simple and popular among Super Turkers), we can help novices increase their wages. For this purpose, we: (i) conducted a survey and data analysis to computationally identify a simple yet common criteria that Super Turkers use for handling transparency tools; (ii) deployed a two-week field experiment with novices who followed this Super Turker criteria to find better work on MTurk. Novices in our study viewed over 25,000 tasks by 1,394 requesters. We found that novices who utilized this Super Turkers' criteria earned better wages than other novices. Our results highlight that tool development to support crowd workers should be paired with educational opportunities that teach workers how to effectively use the tools and their related metrics (e.g., transparency values). We finish with design recommendations for empowering crowd workers to earn higher salaries.

Susumu Saito, Chun-Wei Chiang, Saiph Savage et al.

Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ~150 derived features, and was able to predict the hourly wages of 69.6% of all the tested microtasks within a 75% error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences.)

Anna Kasunic, Chun-Wei Chiang, Geoff Kaufman et al.

In 2013, scholars laid out a framework for a sustainable, ethical future of crowd work, recommending career ladders so that crowd work can lead to career advancement and more economic mobility. Five years later, we consider this vision in the context of Amazon Mechanical Turk (AMT). To understand how workers currently view their experiences on AMT, and how they publicly present and share these experiences in their professional lives, we conducted a survey study with workers on AMT (n=98). The survey we administered included a combination of multiple choice, binary, and open-ended (short paragraph) items gauging Turkers' perceptions of their experiences on AMT within the context of their broader work experience and career goals. This work extends existing understandings of who crowd workers are and why they crowd work by seeking to better understand how crowd work factors into Turkers' professional profiles, and how we can subsequently better support crowd workers in their career advancement. Our survey results can inform the design of better tools to empower crowd workers in their professional development both inside and outside of AMT.

Chun-Wei Chiang, Anna Kasunic, Saiph Savage

Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.

Chun-Wei Chiang, Eber Betanzos, Saiph Savage

Immigrants usually are pro-social towards their hometowns and try to improve them. However, the lack of trust in their government can drive immigrants to work individually. As a result, their pro-social activities are usually limited in impact and scope. This paper studies the interface factors that ease collaborations between immigrants and their home governments. We specifically focus on Mexican immigrants in the US who want to improve their rural communities. We identify that for Mexican immigrants having clear workflows of how their money flows and a sense of control over this workflow is important for collaborating with their government. Based on these findings, we create a blockchain based system for building trust between governments and immigrants. We finish by discussing design implications of our work and future directions.

Chun-Wei Chiang, Caroline Anderson, Claudia Flores-Saviaga et al.

Mobile money can facilitate financial inclusion in developing countries, which usually have high mobile phone use and steady remittance activity. Many countries in Latin America meet the minimum technological requirements to use mobile money, however, the adoption in this region is relatively low. This paper investigates the different factors that lead people in Latin America to distrust and therefore not adopt mobile money. For this purpose, we analyzed 27 mobile money applications on the market and investigated the perceptions that people in Latin America have of such interfaces. From our study, we singled out the interface features that have the greatest influence in user adoption in developing countries. We identified that for the Latin America market it is crucial to create mobile applications that allow the user to visualize and understand the workflow through which their money is traveling to recipients. We examined the significance of these findings in the design of future mobile money applications that can effectively improve the use of electronic financial transactions in Latin America.

Kotaro Hara, Abi Adams, Kristy Milland et al.

A growing number of people are working as part of on-line crowd work, which has been characterized by its low wages; yet, we know little about wage distribution and causes of low/high earnings. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~\$2/h, and only 4% earned more than \$7.25/h. The average requester pays more than \$11/h, although lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is included in our wage calculations, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs future platform design and worker tools to create a more positive future for crowd work.

Vishwajeet Narwal, Mohamed Hashim Salih, Jose Angel Lopez et al.

Bias is a common problem in today's media, appearing frequently in text and in visual imagery. Users on social media websites such as Twitter need better methods for identifying bias. Additionally, activists --those who are motivated to effect change related to some topic, need better methods to identify and counteract bias that is contrary to their mission. With both of these use cases in mind, in this paper we propose a novel tool called UnbiasedCrowd that supports identification of, and action on bias in visual news media. In particular, it addresses the following key challenges (1) identification of bias; (2) aggregation and presentation of evidence to users; (3) enabling activists to inform the public of bias and take action by engaging people in conversation with bots. We describe a preliminary study on the Twitter platform that explores the impressions that activists had of our tool, and how people reacted and engaged with online bots that exposed visual bias. We conclude by discussing design and implication of our findings for creating future systems to identify and counteract the effects of news bias.