Cross-zone traffic is billed per gigabyte, so the annotation that keeps requests inside one availability zone looks like free money:
metadata:
annotations:
service.kubernetes.io/topology-mode: AutoYou apply it, the bill doesn't move, and there's nothing in kubectl describe service that looks like an error. The usual conclusion is that the feature is broken or that the cluster is missing zone labels.
Neither. The controller looked at your service, decided that routing by zone would overload something, and declined. It even said so, in an event nobody reads.
What the controller is actually deciding
The EndpointSlice controller doesn't route anything. It writes hints into EndpointSlices - a note on each endpoint saying "this one is for zone X" - and kube-proxy on each node honours the hints by restricting its endpoint set to the local zone.
It decides who gets a hint from allocatable CPU per zone, and pod counts never enter into it:
The controller allocates a proportional amount of endpoints to each zone. This proportion is based on the allocatable CPU cores for nodes running in that zone.
A zone with twice the CPU is expected to receive twice the traffic, so it should hold twice the endpoints. From that expectation the controller computes, per zone, the smallest endpoint count it would tolerate:
const overloadThreshold float64 = 0.2
desired := ratio * float64(numEndpoints)
minimum := int(math.Ceil(desired * (1 / (1 + overloadThreshold))))Read that as: each endpoint may end up carrying at most 20% more than its fair share. Anything worse and the controller refuses. It sums minimum across zones, and if the total exceeds the number of endpoints you actually have, it gives up and leaves the service on cluster-wide routing.
overloadThreshold is a package-level constant and it isn't exported. There's no flag, no annotation value, no per-service override.
Adding a replica can turn it off
Because minimum is rounded up per zone, the outcome isn't monotonic in replica count. Adding a replica can turn the feature off.
Three zones with equal allocatable CPU, varying the number of ready endpoints:
3 hints
4 refused
5 refused
6 hints
7 hints
8 refused
9 hints
10 hints
11 refused
12 hintsEight is the one that catches teams, because eight is a normal number of replicas and seven works. Walk through it: each zone's fair share is 8/3 = 2.67 endpoints, the minimum is ceil(2.67 / 1.2) = 3, three zones need 9, and you have 8. Refused.
Nine replicas works, which is where the "at least 3 per zone" rule of thumb comes from. That rule is a decent summary of the safe region and a bad description of the boundary, and the boundary is where the surprise lives - an HPA that scales 7 to 8 silently disables zone routing, then re-enables it at 9.
Unequal CPU across zones shifts the whole pattern. On a cluster where one zone runs bigger nodes, the working counts differ from the table above, so treat it as a shape rather than a lookup.
Finding out which case you're in
The controller emits a Warning event on the Service with reason TopologyAwareHintsDisabled, and the message names the specific reason:
InsufficientNumberOfEndpoints- fewer endpoints than zonesMinAllocationExceedsOverloadThreshold- the arithmetic above didn't work outNodesReadyInOneZoneOnly- only one zone has ready nodesNoZoneSpecified- an endpoint's node has no zone label
So the first diagnostic step is not tcpdump:
kubectl get events --field-selector reason=TopologyAwareHintsDisabled -AIf nothing comes back and the hints still aren't there, check whether the hints exist but kube-proxy is ignoring them:
kubectl get endpointslice -l kubernetes.io/service-name=web \
-o jsonpath='{range .items[*].endpoints[*]}{.zone}{" -> "}{.hints}{"\n"}{end}'Empty hints with no event usually means the controller never considered the service at all - the annotation is misspelled, or it's on the Deployment instead of the Service.
The newer field is a different mechanism
spec.trafficDistribution shows up in every discussion of this, and reading it as a tidier spelling of the annotation is a mistake worth avoiding.
They're separate code paths in the same controller. The annotation goes through the capacity heuristic described above. trafficDistribution goes through its own reconciler, whose entire policy is one comment in the source:
update adds a same zone topology hint for all ready endpoints
Every ready endpoint gets a hint for its own zone. No CPU ratios, no overload threshold, no minimum endpoint count, no refusal path. PreferSameNode does the same thing at node granularity.
Which means the migration people describe as cosmetic changes the behaviour in both directions. You gain a feature that actually turns on at 8 replicas. You lose the guard that was protecting a thin zone from taking a full share of traffic with one pod in it.
There's a precedence rule too, and it's silent: the reconciler computes canUseTrafficDistribution as valid field and annotation absent. Leave the old annotation on a Service while adding the new field, and the field is ignored entirely. No event, no warning, no status. The source carries a note that the whole branch goes away once the annotation is deprecated under KEP-4444, so the annotation is on its way out - but while both exist, the old one wins.
The assumption underneath all of it
Both mechanisms allocate endpoints by where capacity is. Neither knows where traffic comes from.
That's fine when request origin roughly tracks capacity, which is the common case for internal service-to-service calls in a balanced cluster. It falls apart when a single zone originates most of the traffic - a batch job, a cron-driven fan-out, an ingress tier that isn't spread the way your workloads are. Then zone-local routing concentrates that load onto one zone's endpoints while the others idle, and the cost saving comes out of your tail latency.
This is also the second half of a two-hop story. externalTrafficPolicy governs the first hop, from the external load balancer to a node, and it has its own skew problem. Topology hints govern the second hop, from the node to a pod. They compose, and they fail independently.
The limits worth knowing
Hints are ignored entirely when internalTrafficPolicy: Local is set on the Service. The two features want the same decision made in different places, and the node-local policy wins.
The controller excludes control-plane nodes from its CPU ratios, ignores unready nodes, and pays no attention to taints or tolerations. A zone whose nodes are tainted so your workload can't schedule there still counts toward that zone's expected share, which is a good way to get refusals you can't explain from the pod distribution alone.
And if any single endpoint is missing a hint while others have them, kube-proxy falls back to all endpoints for that service. Partial hints aren't partially applied - it's all or nothing, per service, on every node.