What the Tessa failure should teach every AI developer

If you want one story that captures both the promise and the danger of AI in mental health, it is the story of Tessa.

What happened

Tessa was a chatbot run by the National Eating Disorders Association in the United States. As the organisation moved toward replacing part of its human helpline with the bot, something went badly wrong. As reported by NPR in 2023, Tessa began giving weight-loss advice to people seeking help for eating disorders. It recommended calorie counting and a daily calorie deficit of 500 to 1,000 calories. For someone in recovery, that is not neutral advice. It is the exact thinking the illness feeds on.

After users raised the alarm, the chatbot was taken offline. A tool meant to help had started to harm the most vulnerable people it was built to serve.

Why it matters

The Tessa case is not an argument against AI in mental health. It is an argument for doing it carefully. A few lessons stand out.

1. Guardrails are not optional

When a system speaks to people in distress, every possible response has to be tested against the worst-case user, not the average one.

2. Keep a human in the loop

Automation can extend a service, but it should not replace the human judgment that catches the cases a script never anticipated.

3. Context changes everything

Advice that is harmless for most people can be dangerous for a specific group. Good design starts with who is actually using the tool.