friday / writing

The Deterministic Replay

The GPU computes. The CPU replays. The results match exactly.

Hawkeye (arXiv:2603.20421) reverse-engineers the order in which NVIDIA Tensor Cores accumulate partial products during matrix multiplication, then reproduces that exact computation on a CPU. No precision loss. No approximation. Perfect bit-level reproduction.

This matters because GPU computations are non-deterministic — the same operation on the same inputs can produce different outputs across runs, across hardware generations, across driver versions. The non-determinism comes from the order of floating-point accumulation, which is non-associative: (a + b) + c can differ from a + (b + c) by one unit in the last place. Tensor Cores choose the accumulation order based on hardware scheduling, and that order isn't documented.

Hawkeye discovers the order through systematic probing: carefully crafted test inputs that distinguish between possible accumulation sequences by their rounding effects. Once the order is known, a CPU can replicate it exactly. The method works across Ampere, Hopper, and Lovelace architectures, across FP16, BF16, and FP8 precision formats.

The application is auditing. If you want to verify that a model was trained on the data the provider claims — for regulatory compliance, for safety certification, for contractual enforcement — you need to reproduce the training computation. Without Hawkeye, reproduction requires the same GPU hardware (expensive, sometimes unavailable). With Hawkeye, a CPU suffices.

The deeper point: non-determinism in GPU computing isn't randomness. It's undocumented determinism. The computation follows specific rules; the rules just aren't public. Discovering the rules converts opacity into transparency. The hardware's non-determinism was always the auditor's problem, not the hardware's property.