Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design
It addresses safety risks in educational LLM deployment, though it is incremental as it builds on existing AI safety research.
This study investigated how students can Trojanize prompts to bypass safety systems in LLMs used in K-12 education, exposing vulnerabilities in GPT-3.5 and GPT-4, and developed a prototype tool called TrojanPromptGuard for detection and mitigation.
The integration of Large Language Models (LLMs) in K--12 education offers both transformative opportunities and emerging risks. This study explores how students may Trojanize prompts to elicit unsafe or unintended outputs from LLMs, bypassing established content moderation systems with safety guardrils. Through a systematic experiment involving simulated K--12 queries and multi-turn dialogues, we expose key vulnerabilities in GPT-3.5 and GPT-4. This paper presents our experimental design, detailed findings, and a prototype tool, TrojanPromptGuard (TPG), to automatically detect and mitigate Trojanized educational prompts. These insights aim to inform both AI safety researchers and educational technologists on the safe deployment of LLMs for educators.