A test fails in CI. The operator must decide: rerun or escalate? If flaky — a transient failure caused by timing, resource contention, or environmental noise — rerunning wastes a slot but finds the answer. If real — a genuine bug — rerunning wastes time while a defect goes uninvestigated. The decision must be fast, because CI pipelines block on it.
Zhu, Wang, Zhai, and colleagues (arXiv:2603.23054) build SCOUT for TiDB, a distributed database where flaky failures are endemic. The framework makes the decision using only information available before the failed run completes: pre-failure telemetry (system metrics, resource usage) and historical data (how often this test flakes, how often this node misbehaves). No post-failure artifacts. No logs from the failed run. The decision is made from context, not from the failure itself.
This constraint — strict causality, no lookahead — is what makes the problem hard and the solution interesting. Most failure triage systems examine the failure: parse the error message, check the stack trace, compare to known patterns. SCOUT can't. It must decide whether a failure is real before looking at it, using only what was true about the system before the failure happened.
The system works: deployed in production with P95 latency of 1.17 ms on CPU, evaluated on 3,680 labeled failures. But the deeper structural point is the label bias. The ground truth — whether a failure was flaky or real — is determined by rerunning. But you can only rerun a finite number of times. A flaky test that fails on all three reruns is labeled “real” even though it's flaky with a high flake rate. The labels themselves are contaminated by the finite budget used to generate them. The calibration must account not just for the distribution of flaky tests but for the policy that decided which tests to label.
The maybe-failure is not a binary with noise. It's a decision under a labeling policy that shapes the ground truth it's trying to learn.