Dawei Xie

Dawei Xie, Khalil Anderson, Tochukwu Eze et al.

Collaboration literacy requires adapting to the evolving demands of group work within complex discussions, making it difficult to develop and assess. Traditional analytics metrics capture behavioral signals while missing the semantic dimensions of how learners approach collaboration and build on each other's ideas. We present Collaboration Literacy through Artifact Reasoning and Augmentation (CLARA), an agentic analytics system that extracts semantic representations from transcripts as analytics artifacts: concept maps representing emergent ideas and relationships, and collaboration assessment characterizing collaboration quality across seven dimensions. While users explore these artifacts through the dashboard, the same artifacts are indexed into distinct vector database collections for agent retrieval and reasoning. This architecture establishes a human-AI common ground where users and AI can operate over shared representations. Evaluation results show that CLARA produces reliable collaboration quality analysis and, owing to the artifacts serving as knowledge infrastructure, improves both retrieval performance and response quality over transcript-only baselines. Our work suggests that AI-produced artifacts may scaffold human interpretation and ground AI reasoning in learning analytics workflows.

Jessica Hullman, Yifan Wu, Dawei Xie et al.

Methods to quantify uncertainty in predictions from arbitrary models are in demand in high-stakes domains like medicine and finance. Conformal prediction has emerged as a popular method for producing a set of predictions with specified average coverage, in place of a single prediction and confidence value. However, the value of conformal prediction sets to assist human decisions remains elusive due to the murky relationship between coverage guarantees and decision makers' goals and strategies. How should we think about conformal prediction sets as a form of decision support? We outline a decision theoretic framework for evaluating predictive uncertainty as informative signals, then contrast what can be said within this framework about idealized use of calibrated probabilities versus conformal prediction sets. Informed by prior empirical results and theories of human decisions under uncertainty, we formalize a set of possible strategies by which a decision maker might use a prediction set. We identify ways in which conformal prediction sets and posthoc predictive uncertainty quantification more broadly are in tension with common goals and needs in human-AI decision making. We give recommendations for future research in predictive uncertainty quantification to support human decision makers.