Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis
This research is significant for professionals in knowledge-intensive fields, particularly those in law, by demonstrating that user training is crucial for realizing productivity gains from generative AI, addressing concerns about reliability and adoption.
This study investigated the impact of targeted user training on the adoption and productive use of generative AI in legal analysis. They found that a 10-minute training intervention significantly increased LLM adoption from 26% to 41% and improved examination performance by 0.27 grade points, equivalent to one-third of a letter grade.
Can targeted user training unlock the productive potential of generative artificial intelligence (GenAI) in professional settings? We investigate this question using a randomized study involving 164 law students completing an issue-spotting examination. Participants were assigned to one of three conditions: no GenAI access, optional access to a large language model (LLM), or optional access accompanied by an approximately ten-minute training intervention. Training significantly increased LLM adoption--the usage rate rose from 26% to 41%--and improved examination performance. Students with trained access scored 0.27 grade points higher than those with untrained access (p = 0.027), equivalent to roughly one-third of a letter grade. By contrast, access to an LLM without training did not improve performance and was associated with shorter answers relative to no access. Using principal stratification, we decompose the overall effect into adoption and effectiveness channels. Point estimates are consistent with training operating primarily by expanding the scope of GenAI use rather than by enhancing effectiveness among existing users, though confidence intervals are wide. Overall, our findings provide evidence that complementary investments in user training are critical for realizing GenAI productivity gains in knowledge-intensive fields where concerns about reliability may inhibit adoption.