An AI system says it's 90% confident. It's actually right 70% of the time when it says this — overconfident. Can humans learn to adjust?
Li and Steyvers (arXiv:2603.22634) ran 200 participants through 50 trials of AI-assisted decisions. The AI provided confidence scores that were systematically miscalibrated: overconfident, underconfident, or inverted (high stated confidence when actually wrong). Over repeated interactions, participants learned to compensate. They improved their accuracy, sharpened their discrimination between correct and incorrect AI outputs, and brought their reliance closer to the AI's actual reliability.
The compensation works through two adjustable parameters: a baseline trust level and a sensitivity to confidence gradations. When the AI is overconfident, participants learn to lower their baseline — they trust the system less overall. When underconfident, they raise it. The adjustment follows a Rescorla-Wagner learning rule: the difference between expected and actual outcomes drives incremental updates to the trust parameters.
But the inverted condition breaks this. When the AI's confidence signal is negatively correlated with accuracy — when “90% confident” means “probably wrong” — participants struggle. They improve somewhat, but never fully compensate. The learning mechanism can adjust magnitude (how much to trust) but not direction (whether confidence means reliability or its opposite). There's an inversion barrier.
The structural lesson is about the format of the signal. Overconfidence and underconfidence preserve the direction: high confidence still means relatively more likely to be right, just not as likely as stated. The adjustment is a scaling operation — compress or expand the confidence axis, but keep its orientation. Inversion requires flipping the axis entirely, and the Rescorla-Wagner update rule, which adjusts by proportional nudges, can't achieve a sign change through incremental steps.
Humans can learn to distrust an unreliable system. They can learn to trust a humble one. What they can't easily learn is that confidence is anti-correlated with accuracy — that the system is most dangerous when it seems most sure.