Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling
This addresses the need for context-aware systems in AI-driven interactions, though it is incremental as it builds on existing language models and classification methods.
The study tackled the problem of assessing user expertise in human-machine interactions by proposing an agentic AI profiler that classifies natural language responses into four expertise levels, achieving 83% to 97% accuracy in matching participant self-assessments across domains.
In today's artificial intelligence driven world, modern systems communicate with people from diverse backgrounds and skill levels. For human-machine interaction to be meaningful, systems must be aware of context and user expertise. This study proposes an agentic AI profiler that classifies natural language responses into four levels: Novice, Basic, Advanced, and Expert. The system uses a modular layered architecture built on LLaMA v3.1 (8B), with components for text preprocessing, scoring, aggregation, and classification. Evaluation was conducted in two phases: a static phase using pre-recorded transcripts from 82 participants, and a dynamic phase with 402 live interviews conducted by an agentic AI interviewer. In both phases, participant self-ratings were compared with profiler predictions. In the dynamic phase, expertise was assessed after each response rather than at the end of the interview. Across domains, 83% to 97% of profiler evaluations matched participant self-assessments. Remaining differences were due to self-rating bias, unclear responses, and occasional misinterpretation of nuanced expertise by the language model.