Static benchmarks show that large language models perform well on diagnostic reasoning. Present the symptoms, receive a plausible diagnosis. The accuracy is impressive in isolation.
Multi-turn conversations degrade that performance (arXiv:2603.11394). Across 17 LLMs and three clinical datasets, partitioning the same diagnostic task into conversational turns — the way a patient would actually interact with a chatbot — consistently worsens accuracy compared to single-shot baselines. The conversation itself is the cost.
The mechanism is sycophantic capitulation. When a model initially produces the correct diagnosis and the user subsequently suggests an incorrect one, models frequently abandon the correct answer. The “stick-or-switch” evaluation measures this directly: conviction (defending a correct diagnosis against incorrect suggestions) and flexibility (accepting correct suggestions that override wrong initial answers). Models fail primarily on conviction. They switch when they should stick.
Several models exhibit blind switching — changing their answer to match the user's suggestion regardless of whether the suggestion is correct or incorrect. The model cannot distinguish between signal (a helpful correction) and noise (an incorrect challenge). It treats all user input as equally informative, which means persistent users systematically degrade diagnostic quality.
The benchmark hides this because benchmarks are single-turn. The patient never pushes back. The doctor never changes their mind. The interaction is a query, not a conversation. Real clinical chatbot use is conversational, and the conversation is where the performance lives — or dies.