A distributed training job asks for eight GPU pods. Seven schedule within seconds. The eighth sits in Pending - one node short of capacity somewhere in the cluster - and stays there. Nothing crashes. Nothing errors. The training process on all seven running pods just hangs, because the all-reduce step that synchronizes gradients across the job is waiting for a peer that never showed up.
Seven GPUs, burning money, doing nothing, for however long it takes someone to notice and kill the job.
The default Kubernetes scheduler did exactly what it was built to do: schedule pods, one at a time, as capacity allows. That's precisely the problem.
Why this isn't a podAffinity problem
The instinct is to reach for podAffinity or topologySpreadConstraints to keep the job's pods close together. Neither one touches the actual failure. Both operate per-pod, evaluated independently as each pod is scheduled - they can influence where a pod lands, but nothing about them makes eight pods land together or not at all. The scheduler still admits each pod as its own decision, and a job that needs all eight can still end up with seven.
Distributed training, MPI jobs, Spark executors that all need to see each other before work starts - none of these have a "partial success" mode. A job that gets 7 of 8 pods isn't 87% done. It's 0% done, at full cost.
Gang scheduling: the actual fix
Gang scheduling flips the admission decision from per-pod to per-job: either every pod in the group gets scheduled together, or none of them do, and the ones that would've scheduled successfully get held back too. HPC batch schedulers have handled this exact problem for decades; Kubernetes borrowed the concept late, and it shows up now as "all-or-nothing" semantics in a handful of projects: Kueue, Volcano, Apache YuniKorn.
Kueue is the one worth knowing first: an official Kubernetes SIG-Scheduling project (kubernetes-sigs/kueue), not some third-party fork of the scheduler, currently at v0.13.4 and shipping regularly.
Links
How Kueue actually enforces it
Kueue sits in front of the default scheduler as an admission layer, rather than replacing it. Jobs go into a LocalQueue, a namespace-scoped object your team submits work to, which references a ClusterQueue that owns the actual quota and fair-sharing rules across the cluster. Kueue holds a job back until it's confident the whole group can be admitted, then releases all of its pods to the underlying scheduler together.
The enforcement mechanism is waitForPodsReady: a cluster-wide, timeout-based check. Once Kueue admits a workload, it watches until every pod in it reports ready. If the timeout passes before that happens - because a node disappeared, or capacity got contended by something else - Kueue evicts the whole workload and requeues it, rather than leaving it half-running and silently burning resources. That's the mechanism that turns "7 of 8, hanging forever" into "0 of 8, retried automatically."
Links
Topology awareness: gang scheduling isn't enough on its own
Getting all eight pods scheduled somewhere solves the all-or-nothing problem. It doesn't solve the next one: distributed training over NVLink or 400G Ethernet is latency-sensitive enough that where those eight pods land relative to each other changes throughput by a lot, not a little.
Kueue's topology-aware placement understands a zone-over-rack-over-host hierarchy and bin-packs a job's pods onto adjacent nodes rather than scattering them wherever capacity happens to be free. Gang scheduling gets you eight pods running together. Topology awareness gets you eight pods running together fast.
This isn't only an ML problem
The specific example is GPU training because that's where the pain is loudest right now, but the underlying failure mode - a job whose pods have to arrive together to do useful work at all - applies to any tightly-coupled distributed system: MPI jobs, Spark executor pools, anything with a synchronization barrier before real work starts. If your workload has a moment early on where every worker waits for every other worker, the default scheduler's per-pod admission model is working against you, not for you.
Questions? Feedback? Reply to this email. I actually read them.
- Ilia