Agentic AI for Substance Use Education: Integrating Regulatory and Scientific Knowledge Sources
This work addresses the need for scalable, personalized, and up-to-date substance use education, but the evaluation is small-scale (5 experts, 30 questions) and the novelty is incremental.
The authors built an agentic AI web application that integrates Drug Enforcement Administration records with peer-reviewed literature to provide real-time substance use education. Expert evaluation showed mean ratings of 4.18-4.35 across four criteria (factual accuracy, citation quality, contextual coherence, regulatory appropriateness) with substantial inter-rater agreement (Cohen's kappa = 0.78).
The delivery of traditional substance education has remained problematic due to challenges in scalability, personalization, and the currency of information in a rapidly evolving substance use landscape. While artificial intelligence (AI) offers a promising frontier for enhancing educational delivery, its application in providing real-time, authoritative substance use education remains largely underexplored. We built an agentic-based AI web application that combined Drug Enforcement Administration records with peer-reviewed literature in real-time to provide transparent context-sensitive substance use education. The system uses retrieval-augmented generation with a carefully filtered corpus of 102 documents and dynamic PubMed queries. Document storage was semantically chunked and placed in a vector representation in order to be easily retrieved. We conducted an expert evaluation study in which a panel of five subject matter experts generated 30 domain-specific questions, and two independent raters assessed 90 system interactions (30 primary questions plus two contextual follow-ups each) using a five-point Likert scale across four criteria: factual accuracy, citation quality, contextual coherence, and regulatory appropriateness. Mean ratings ranged from 4.18 to 4.35 across the four criteria (overall category range: 4.05-4.52), with substantial inter-rater agreement (Cohen's kappa = 0.78). These findings suggest that agentic AI architectures integrating authoritative regulatory sources with real-time scientific literature represent a promising direction for scalable, accurate, and verifiable health education delivery, warranting further evaluation through longitudinal user studies.