Rachith Aiyappa

Rachith Aiyappa, Jisun An, Haewoon Kwak et al.

ChatGPT, the first large language model (LLM) with mass adoption, has demonstrated remarkable performance in numerous natural language tasks. Despite its evident usefulness, evaluating ChatGPT's performance in diverse problem domains remains challenging due to the closed nature of the model and its continuous updates via Reinforcement Learning from Human Feedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study of the task of stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.

Matthew R. DeVerna, Francesco Pierri, Rachith Aiyappa et al.

To analyze the flow of information online, experts often rely on platform-provided data from social media companies, which typically attribute all resharing actions to an original poster. This obscures the true dynamics of how information spreads online, as users can be exposed to content in various ways. While most researchers analyze data as it is provided by the platform and overlook this issue, some attempt to infer the structure of information cascades. However, the absence of ground truth about actual diffusion cascades makes it impossible to verify the efficacy of these efforts. We propose a novel parametric reconstruction approach and use it to investigate how overlooking cascade reconstruction distorts analyses of social influence, community detection, and information diffusion. Two case studies involving data from Twitter and Bluesky reveal that cascade inference significantly impacts the identification of both influential users and communities, therefore affecting downstream analyses in general. Analysis of the diffusion of over 40,000 true and false news stories on Twitter reveals that the assumptions made during the reconstruction procedure drastically distort both microscopic and macroscopic properties of cascade networks. This work highlights the challenges of studying information spreading processes on complex networks and has significant implications for the broader study of digital platforms.