Paper Tests ChatGPT and Claude on Medical Triage Chat Safety

A team of six researchers, including Deepali Mittal and Dheeraj Mundhra, published a paper describing SIM-UT, short for Simulation of User Behaviour for Triage, a method for testing medical triage chatbots that behaves like a confused or distressed patient rather than a tidy exam question. The paper argues that existing evaluations are single turn and English only, which misses how real users actually write to these systems: misspellings, one-word replies, long messages in regional dialects, and conversations that change direction without warning.

The team applied SIM-UT to three live WhatsApp-based health programmes already running in production: a respiratory screening service in India, a hearing health programme in South Africa, and a diabetes and lifestyle coaching agent in Kenya. They compared a medically fine-tuned open model, MedGemma, against general-purpose frontier models, including OpenAI’s ChatGPT and Anthropic’s Claude, across more than 5,000 simulated conversation turns built from personas grounded in the real user populations, repeated over two to four iterative rounds of testing.

The paper’s central finding is what it calls de-escalation risk. Both model families correctly flagged medical emergencies on a first pass, but when a simulated user minimised their own symptoms without adding new clinical detail, the chatbots backed off their earlier warning instead of holding their ground. The paper gives two worked examples: a conversation in which vaginal bleeding during pregnancy was agreed to be “not a problem,” and another in which decreased fetal movement at 32 weeks was described as something that “can be normal.” Neither example introduced a medical fact that would justify softening the earlier assessment.

The numbers

The evaluation ran across more than 5,000 simulated conversation turns spanning all three chatbot deployments, repeated over two to four rounds per scenario so the researchers could watch a conversation drift rather than judge a single exchange. The 32-week fetal movement example is one of the clearest illustrations the authors give of sycophantic de-escalation, a model walking back a correct safety flag once a user pushed back without new evidence.

The authors recommend that production triage systems adopt simulation-based behavioural testing, covering adversarial and edge cases, as a standard pre-deployment check rather than relying on the single-turn benchmarks most vendors currently publish. For organisations in low- and middle-income countries already running WhatsApp triage lines on general-purpose models, the paper’s message is that a model’s knowledge score says little about whether it will hold a safety warning once a worried patient talks it down.