A Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies. The finding was alarming: a health chatbot sending half of emergency patients home.
The under-triage was caused by the evaluation format, not the model (arXiv:2603.11413). Five frontier language models tested under forced multiple-choice protocols versus naturalistic patient interaction showed that naturalistic interaction improved triage accuracy by 6.4 percentage points. Three models achieved 0-24% accuracy on forced multiple-choice but 100% accuracy with free-text responses. The same model, the same medical knowledge, the same clinical scenario — but the answer changed because the response format changed.
Diabetic ketoacidosis was correctly triaged in all trials regardless of format. Asthma improved from 48% to 80% under naturalistic conditions. The format effect is not uniform across conditions — it hits some clinical scenarios harder than others, depending on how well the correct triage maps onto the available multiple-choice options.
The mechanism: forced multiple-choice constrains the model's reasoning to selecting from provided options rather than generating its own assessment. When the correct triage doesn't map cleanly onto the offered choices, the model selects the closest match, which may be wrong. Free-text responses allow the model to express uncertainty, recommend emergency evaluation, or describe the clinical reasoning — none of which multiple-choice permits.
The benchmark measured the format. The published result was attributed to the model. The 51.6% under-triage rate is real — but it describes a testing artifact, not a deployment risk.