Concerns on Bias in Large Language Models when Creating Synthetic Personae
It addresses ethical risks in HCI research by identifying bias in LLMs, but it is incremental as it builds on existing concerns without major new solutions.
The paper examines bias in large language models when generating synthetic personae for HCI research, finding evidence of bias through vignette-based experiments and highlighting the need for rigorous testing.
This position paper explores the benefits, drawbacks, and ethical considerations of incorporating synthetic personae in HCI research, particularly focusing on the customization challenges beyond the limitations of current Large Language Models (LLMs). These perspectives are derived from the initial results of a sub-study employing vignettes to showcase the existence of bias within black-box LLMs and explore methods for manipulating them. The study aims to establish a foundation for understanding the challenges associated with these models, emphasizing the necessity of thorough testing before utilizing them to create synthetic personae for HCI research.