Run a hundred A/B tests. Ship the winners. The problem: the experiments you select for deployment are the ones most likely to be overestimating their effect. This is the Winner's Curse — selection bias baked into the decision process.
Mudd, Zaidi, Friedberg, Gorbachev, Choubey, and Nassif from Meta Platforms quantify the damage and offer a fix. The curse is worst in low-power regimes — exactly the conditions most common in practice, where sample sizes are limited by cost, time, or traffic. Experiments that barely clear the significance threshold are disproportionately likely to have gotten lucky. The effect estimate you're selecting on is the same estimate you're using to project impact. The selection and the measurement are the same number.
Their solution — Bayesian Hybrid Shrinkage — introduces experiment-specific local shrinkage rather than uniform shrinkage. Each experiment is pulled toward a data-driven prior by an amount that depends on its own characteristics, not just the population. This matters because uniform shrinkage penalizes strong experiments and under-corrects weak ones. The hybrid approach respects the heterogeneity.
The closed-form inference strategy is designed for production scale — Meta's experimentation platform runs at volumes where iterative methods are too slow. The practical constraint shapes the statistical method, not the other way around.
The broader lesson extends past A/B testing. Any decision process that selects on a measured quantity and then uses that same measurement for downstream planning inherits the curse. Hiring decisions based on interview scores. Grant funding based on proposal ratings. The experiments that look best are the ones where the measurement error happened to point up. Shrinkage toward a data-driven prior is the general correction — but it requires admitting that your best-looking result is probably not as good as it looks.