nginx proxy_pass: the variable that costs you the upstream block
The standard fix for stale pod IPs trades one visible failure for three that never show up in your logs, and resolve makes it unnecessary
The 502s came in bursts, always right after a deploy, always for about ninety seconds. nginx was proxying to a headless service, and it had resolved those pod IPs once at startup. The pods behind them had been replaced. nginx didn't know and had no reason to find out.
The fix everybody reaches for is one line, and it works:
set $backend "app.default.svc.cluster.local";
proxy_pass http://$backend;The 502s stop. The bill arrives later.
Why the one-liner works
When proxy_pass takes a literal address, nginx resolves it once, at configuration load, and caches the result for the process lifetime. No TTL, no re-resolution, no amount of waiting will change it. Only a reload will.
Put a variable in there and the rule changes. From the ngx_http_proxy_module docs:
> Parameter value can contain variables. In this case, if an address is specified as a domain name, the name is searched among the described server groups, and, if not found, is determined using a resolver.
Two branches, and which one you land on decides everything that follows.
If the resolved string matches the name of an upstream block you've declared, nginx uses that group - and you're back to static peers, because the group's addresses were resolved at load time. The workaround does nothing. People hit this, conclude that variables in proxy_pass don't work, and go looking for a plugin.
If it doesn't match any declared group, nginx hands the name to resolver and honours the DNS TTL. That's the branch that fixes the 502s, and it's the one you get by writing out a full service FQDN, since nobody names an upstream block app.default.svc.cluster.local.
What the second branch costs
You're no longer proxying to an upstream group. You're proxying to whatever a DNS answer produced on this request. Everything the upstream block was doing goes away with it, and none of it announces itself.
Keepalive is the expensive one. The keepalive directive lives inside upstream; with no group there's no connection cache, and you pay a fresh TCP handshake - plus a TLS handshake, if the hop is encrypted - per request. This gap got wider in March 2026: nginx 1.29.7 switched upstream connections to HTTP/1.1 with keepalive enabled by default, 32 connections per worker. Upstream blocks now get connection reuse with no configuration at all. The variable path still gets none, so an upgrade that improved everyone else's latency silently widened the distance between you and them.
Load balancing goes too. least_conn, hash, weights - all upstream directives. What you get instead is nginx trying resolved addresses in the order the resolver returned them, which is round-robin at best and sticky at worst, depending on how your DNS server orders records.
Passive health checking goes. max_fails and fail_timeout mark a peer down after repeated failures and stop sending it traffic. Without a group there's no peer state to mark, so a pod that's accepting connections and failing every request keeps getting its share until DNS stops returning it.
And URI handling changes shape. With a literal proxy_pass that has a path component, nginx rewrites the request URI relative to the location prefix. With a variable it doesn't, so a config that was silently relying on that rewrite starts sending a different path upstream than it used to.
The directive that makes the trade unnecessary
resolve on an upstream server has existed since 1.5.12, but until November 2024 it was NGINX Plus only, which is why so much of the advice online predates it and never mentions it. It landed in open source in 1.27.3 (mainline), then 1.28.0 (stable, April 2025).
upstream app {
zone app 64k;
server app.default.svc.cluster.local resolve;
keepalive 32;
}
resolver 10.96.0.10 valid=10s ipv6=off;
location / {
proxy_pass http://app;
}nginx re-resolves the name in the background on DNS TTL and rewrites the peer list in place, without a reload. You keep the group, so you keep keepalive, the balancing method, and max_fails. It also lets nginx start when the name doesn't resolve yet, marking the peer down instead of refusing to load - which removes the other classic failure, where a restart during a DNS blip takes the proxy down entirely.
Two requirements, and both are hard errors if you skip them. The group must be in shared memory, so zone is mandatory - peers live in one place where every worker sees updates. And resolver must be configured, in the http block or in the upstream block itself. In Kubernetes point it at the cluster DNS service address and set valid= explicitly rather than trusting the record TTL, which is often 30 seconds and will feel slow during a rollout.
The general shape of this bug
The pattern isn't an nginx thing. A reverse proxy has to decide when a name becomes an address, and every proxy offers a fast path that resolves once and a slow path that resolves continuously, with different feature sets hanging off each.
Envoy calls them STRICT_DNS and LOGICAL_DNS, and the difference isn't what most people assume from the names: STRICT_DNS keeps a connection pool per resolved address, LOGICAL_DNS keeps one logical host and only uses the first address returned, which makes it cheaper and much worse at spreading load. HAProxy needs resolvers plus a server-template for the same effect, and a plain server line with a hostname is resolved at startup exactly like nginx.
If you take one thing: when a proxy suddenly starts tracking DNS changes, check what it stopped doing in exchange. There's almost always something, and it's almost never in the same paragraph of the docs.
The limits worth knowing
resolve follows DNS, and DNS isn't a health signal. A pod that's Terminating stays in the endpoints list until the endpoint controller removes it, and your resolver caches that answer for valid=. Connection draining still needs preStop and a sane terminationGracePeriodSeconds on the pod side.
For most Kubernetes services this whole problem is avoidable. A normal ClusterIP gives you one stable virtual IP that never changes, and kube-proxy or the CNI handles the fan-out. The stale-IP problem is specific to headless services, where you deliberately asked for pod IPs. If you can't articulate why you need clusterIP: None, the cheapest fix is to stop using it.