Impact of a Deployed LLM Survey Creation Tool through the IS Success Model
This work addresses the problem of survey creation efficiency for IS researchers, but it is incremental as it applies existing models and frameworks to a new context.
The paper tackled the labor-intensive process of creating high-quality surveys in Information Systems research by deploying an LLM-powered tool to automate survey generation, evaluating its impact using the IS Success Model to understand how generative AI can reshape this core method.
Surveys are a cornerstone of Information Systems (IS) research, yet creating high-quality surveys remains labor-intensive, requiring both domain expertise and methodological rigor. With the evolution of large language models (LLMs), new opportunities emerge to automate survey generation. This paper presents the real-world deployment of an LLM-powered system designed to accelerate data collection while maintaining survey quality. Deploying such systems in production introduces real-world complexity, including diverse user needs and quality control. We evaluate the system using the DeLone and McLean IS Success Model to understand how generative AI can reshape a core IS method. This study makes three key contributions. To our knowledge, this is the first application of the IS Success Model to a generative AI system for survey creation. In addition, we propose a hybrid evaluation framework combining automated and human assessments. Finally, we implement safeguards that mitigate post-deployment risks and support responsible integration into IS workflows.