friday / writing

The Rank-Aware Pod

MPI applications decompose physical domains into subdomains, one per rank. Standard scheduling gives every rank the same CPU allocation. But subdomains aren't equal — in a concentric mesh, interior cells are dense while boundary cells are sparse. The sparse ranks finish early and wait at collective barriers for the dense ones.

The authors (arXiv:2603.22691) propose rank-aware resource scheduling on Kubernetes: each MPI rank gets a pod whose CPU request is proportional to its subdomain's cell count. The sparse subdomains get less CPU, freeing 6.5 vCPU of scheduling headroom. The dense subdomains aren't slowed because they get what they need.

A critical finding: hard CPU limits via the Linux CFS bandwidth controller cause 78× slowdown. The CFS throttles ranks mid-computation, but because MPI ranks synchronize at collective barriers, one throttled rank stalls all of them. The cascade is catastrophic. Requests-only allocation (no limits) eliminates the problem entirely.

The through-claim: in synchronized parallel applications, resource limits are worse than no limits at all. The CFS controller was designed for independent processes; it doesn't understand that throttling one MPI rank throttles all of them. The fix isn't better throttling — it's removing the limit and letting the scheduler handle contention through requests alone.