AI systems that detect rare events — fraud, manufacturing defects, medical conditions — depend on human-labeled training data. When the target event is rare, the annotators develop a cognitive bias: they stop expecting to see positives. The prevalence effect is well-documented in radiology, airport security, and quality inspection. It increases missed detections precisely when detection matters most.
A field experiment on a medical crowdsourcing platform quantifies the effect and tests mitigations (arXiv:2603.11511). The true prevalence of the target (blast cells in blood smears) was held constant at 20% in unlabeled data, while the prevalence of gold-standard feedback examples was manipulated: 20% versus 50%. Higher feedback prevalence — showing annotators more positive examples in their calibration stream — reduced rare-event misses.
The response method also mattered. Probability estimation outperformed binary classification. When annotators express confidence (70% likely a blast cell) rather than making a binary call (blast/not-blast), the resulting labels carry more information and are better calibrated.
The gains survived downstream. Convolutional neural networks trained on the recalibrated annotations performed better on unseen data — the improvement in labeling quality propagated through the entire pipeline to the deployed model.
The prevalence effect is a systematic distortion in the training data, not random noise. Treating it as noise (averaging over annotators) doesn't remove it. Treating it as bias (recalibrating against known prevalence) does. The ghost of low prevalence haunts the training data unless explicitly exorcised.