Issue #031 - Ambient mesh: the sidecar was a five-year detour
Istio's node proxy opens its listening sockets inside your pod's network namespace, and it holds a workload certificate for every service account scheduled there
Ambient mode went GA in Istio 1.24, in November 2024. That's twenty-one months ago. Istio is on 1.30.3 now, and somewhere in that stretch the question quietly flipped: it stopped being "should we try ambient" and became "why are we still injecting sidecars".
I covered ambient in issue #002 back in January, at the level of the sidecar tax - one Envoy per pod, fifty to a hundred megabytes each, startup ordering problems. That framing is fine and it's also the least interesting thing about the architecture. The resource number is a consequence. The mechanism underneath is stranger than "we moved the proxy to the node".
🏗️ Architectural Pattern: how a node proxy pretends to be a sidecar
The file descriptor trick
The obvious way to build a node-level proxy is to run it on the node, in the host network namespace, and route pod traffic to it. That design has a well-known problem: once the packet leaves the pod, you've lost the pod. Source addresses get rewritten, the proxy has to reconstruct which workload sent what, and every policy decision downstream depends on that reconstruction being right.
Ambient does something else. When a pod joins an ambient-enabled namespace, the istio-cni node agent enters the pod's network namespace, installs redirection rules there, and then hands ztunnel a file descriptor referring to that namespace over a Unix socket.
Ztunnel uses the descriptor to open listening sockets inside the pod's network namespace. One logical proxy instance per pod, sockets that live in the pod's own netns, a process that lives on the node.
So from the pod's point of view the proxy is local. From the node's point of view there's one Rust process instead of a hundred Envoys. The pod identity never has to be reconstructed, because the traffic never left the pod's namespace unattributed.
Three ports do the work, and they're worth knowing when you read a ss output at 2am:
15001 outbound, before HBONE encapsulation
15006 inbound plaintext
15008 inbound HBONEInbound TCP is redirected to 15008 or 15006 depending on whether it arrived encrypted. Outbound goes to 15001 and leaves the node wrapped in HBONE, which is HTTP CONNECT over mTLS - a tunnel that carries the original destination as connection metadata rather than as a rewritten address.
The identity problem, and the fix
In the sidecar model, the Envoy holding a workload's certificate ran in that workload's pod. The blast radius of a compromised proxy was one pod, and the certificate it held was its own.
Ztunnel holds a certificate for every unique service account among the pods on its node. It has to - it terminates mTLS on their behalf. A node running forty pods across twelve service accounts has one process holding twelve workload identities.
The guardrail is on the CA side. Ztunnel authenticates to the control plane as itself and then requests certificates for other identities, and the CA enforces that it may only do that for identities actually scheduled on that node. Requests for identities not running there are rejected.
That's a sound design and it is a genuinely different threat model from a sidecar. It's node-scoped rather than pod-scoped, and the thing standing between a compromised ztunnel and a cluster-wide identity grab is a control-plane check rather than the absence of the key material. If your security review asked "what changed", that's the answer, and it's better stated plainly than discovered later.
Where waypoints sit
Ztunnel does L4 and only L4. mTLS, L4 authorization, TCP-level telemetry. It doesn't parse HTTP headers, by design - that's how it stays small and how it stays Rust.
Anything at L7 needs a waypoint, which is a real Envoy, deployed as a normal Deployment for a namespace or a service account. The path becomes:
source pod -> source ztunnel -> waypoint -> destination ztunnel -> destination podFour hops where a sidecar mesh had two. That's the trade you're making for the resource savings, and whether it matters depends entirely on how much of your traffic actually needs L7 treatment.
Links
🆚 The Showdown: what the sidecar was still doing for you
The migration guides are cheerful. The list below is the one I'd want before committing, because every item on it is a thing that worked yesterday and needs a decision today.
Per-workload EnvoyFilter and WASM. In sidecar mode you could attach a filter to one workload, because that workload had its own Envoy. Ambient has no per-pod Envoy to attach to. Filters either become global or move into a waypoint, and a waypoint serves a namespace or a service account rather than a single pod. Teams with one bespoke filter for one service feel this immediately.
L7 telemetry by default. A sidecar emitted HTTP metrics, logs and traces for everything passing through it, because everything passing through it was already parsed. Ztunnel gives you L4: bytes, connections, TCP-level authorization decisions. Your request-rate-by-response-code dashboard goes quiet for any workload without a waypoint. This is the single most common surprise, and it surprises people after the migration rather than during it.
Locality-aware routing. A sidecar carried locality information and preferred endpoints in its own zone without being asked. In ambient, traffic that goes through a waypoint goes to wherever that waypoint is, so a waypoint in the wrong zone turns a same-zone call into a cross-zone one - with the latency and the inter-AZ transfer bill that implies. The fix is deploying a waypoint per zone, which is fine, and which nobody plans for on the first pass.
VirtualService. Gateway API is the preferred routing surface in ambient. VirtualService support is alpha there. If your routing rules are years of accumulated VirtualService, that's a translation project, not a label change.
Multi-cluster. Stable in sidecar mode. Available in ambient with limitations that move release to release, which is the polite way of saying check the notes for your exact version rather than trusting a blog post.
And the resource claim deserves an honest reading. "Up to ninety percent" is real arithmetic under the right conditions: it's a ratio between one node proxy and N sidecars, so it scales with pod density and it shrinks the moment you deploy waypoints, because a waypoint is an Envoy again. A cluster with thirty pods a node and L7 policy everywhere saves much less than a cluster with a hundred pods a node running L4 mTLS only. Both are ambient. The savings differ by an order of magnitude.
The failure mode nobody mentions
A sidecar failing takes down one pod. Ztunnel is a DaemonSet, and a ztunnel restart interrupts every meshed pod on that node.
In practice this is mostly fine, because ztunnel restarts are fast and connections re-establish. It's still a different operational shape, and it belongs in the same conversation as the identity scoping: you traded per-pod failure domains for per-node ones, in exchange for not having to restart every application pod to upgrade the mesh.
Links
👮 The Policy: migrating without a flag day
The good news is that the migration is genuinely incremental, and not in the marketing sense. Sidecar pods and ambient pods coexist in one mesh. That means the unit of migration is a namespace, and the rollback is removing a label.
A sequence that works:
Start by turning on L4 for one namespace and changing nothing else. Label it, and you get mTLS and L4 authorization with no pod restarts and no waypoint. This step costs almost nothing in resources and it's where most of the security benefit lives. If you stop here permanently for most namespaces, that's a legitimate end state rather than a half-migration.
Then work out which services need L7. Not which ones have an Envoy today - which ones use a feature that requires parsing HTTP. Header-based routing, JWT claims in authorization policy, retries and timeouts on HTTP semantics, request-level telemetry that someone alerts on. Usually a much shorter list than the number of workloads currently running sidecars.
Deploy waypoints for those and size them like anything else you own. A waypoint is a Deployment: it needs requests, limits, an HPA if the traffic warrants it, and a copy per zone if you care about locality. Treating it as invisible infrastructure is how you get a mesh-wide latency regression out of one under-provisioned pod.
Sort out the dashboards before the cutover, not after. Know which panels are fed by L7 metrics from sidecars, and decide for each one whether it moves to a waypoint or gets rebuilt on L4 signals. Leaving it until afterwards buys you a window where nobody can see anything, and that window is usually when somebody asks whether the migration broke something.
The honest part
This isn't free and it isn't instant. The resource win is real, the operational win of not restarting pods to upgrade the mesh is real, and both are paid for with an extra hop for L7 traffic, a node-scoped identity surface, and a translation project for anyone deep in VirtualService or per-workload filters.
There's also a version of this decision that ends with "neither". If you're running a mesh for mTLS and nothing else, and you're on a cloud CNI that can do transparent encryption between nodes, the honest comparison isn't sidecar against ambient - it's mesh against no mesh. Ambient makes the mesh cheap enough that the question is worth asking again, which is not the same as answering it yes.
Links
What the detour was actually for
Calling the sidecar a five-year detour is unfair in one direction and accurate in the other.
It was unfair because the sidecar proved the model. Transparent mTLS, policy that doesn't live in application code, telemetry nobody had to instrument - all of that had to be demonstrated before anyone would accept a new component in the datapath at all, and the sidecar demonstrated it in the only way available at the time. The kernel primitives ambient depends on weren't there in 2018.
It's accurate because the cost model was wrong from the start, and everyone knew it. One proxy per pod is O(pods) for a job that's O(nodes) plus a bit of L7. We spent five years paying a linear tax for a constant-ish problem, and building tooling to make the tax bearable: injection webhooks, startup ordering, the whole holdApplicationUntilProxyStarts genre of workaround.
The pattern generalises past meshes, and it's the thing I'd take away if I only took one. When a piece of infrastructure needs a lot of tooling to make its cost model tolerable, the tooling is a signal about the cost model, not a solution to it. Sidecar injection webhooks were a very good answer to a question that shouldn't have been asked.
I went and counted our own sidecars after Istio 1.30 shipped. Two hundred and eleven Envoys, of which eleven are attached to workloads that use a single L7 feature. The other two hundred are doing mTLS.
Questions? Feedback? Reply to this email. I actually read them.
Ilia