Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants
This work addresses safety risks for vulnerable users in health-adjacent AI systems, offering an incremental improvement through lightweight, auditable safeguards.
The paper tackled the problem of unsafe AI in healthcare by analyzing the Tessa chatbot incident and proposing a modular safety middleware that combines lexical gates and an LLM policy filter, achieving perfect interception of unsafe prompts with baseline cost and latency in synthetic evaluations.
In 2023, the National Eating Disorders Association's (NEDA) chatbot Tessa was suspended after providing harmful weight-loss advice to vulnerable users-an avoidable failure that underscores the risks of unsafe AI in healthcare contexts. This paper examines Tessa as a case study in absent safety engineering and demonstrates how a lightweight, modular safeguard could have prevented the incident. We propose a hybrid safety middleware that combines deterministic lexical gates with an in-line large language model (LLM) policy filter, enforcing fail-closed verdicts and escalation pathways within a single model call. Using synthetic evaluations, we show that this design achieves perfect interception of unsafe prompts at baseline cost and latency, outperforming traditional multi-stage pipelines. Beyond technical remedies, we map Tessa's failure patterns to established frameworks (OWASP LLM Top10, NIST SP 800-53), connecting practical safeguards to actionable governance controls. The results highlight that robust, auditable safety in health-adjacent AI does not require heavyweight infrastructure: explicit, testable checks at the last mile are sufficient to prevent "another Tessa", while governance and escalation ensure sustainability in real-world deployment.