Gateway API request mirroring: a shadow copy of prod traffic
HTTPRoute RequestMirror filter, fire-and-forget copies, idempotency traps, fraction-based sampling
The rewrite had passed every test we had. Six months of porting a payment-adjacent service from its legacy runtime, a test suite everyone trusted, load tests green, staging green. The one thing nobody could answer was the only question that mattered: what happens when it meets the traffic we can't imagine - the malformed headers, the retries from that one mobile client, the requests that only show up on the last Friday of the month. Staging never has those. Prod has nothing but.
Request mirroring is the answer to that specific fear, and in Gateway API it's a first-class filter rather than a vendor annotation. The gateway takes each incoming request, sends it to your real backend as usual, and sends a copy to a shadow backend. The response from the shadow gets dropped on the floor. Users never see it, latency doesn't grow because the copy is fire-and-forget, and your v2 gets a full day of real production traffic without owning a single response.
The filter itself
Mirroring lives in HTTPRoute, in the filters list of a rule:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: store-route
spec:
parentRefs:
- name: my-gateway
hostnames:
- "store.example.com"
rules:
- backendRefs:
- name: store-v1
port: 8080
filters:
- type: RequestMirror
requestMirror:
backendRef:
name: store-v2-shadow
port: 8080Traffic flows to store-v1 and answers come from store-v1. The copy goes to store-v2-shadow, whose responses the gateway must ignore - that's in the spec, not an implementation courtesy. Because it's Gateway API rather than an Ingress annotation, the same manifest works whether the underlying data plane is Envoy Gateway, Istio, Cilium or anything else that implements the filter. RequestMirror sits at "extended" conformance, so check your implementation's support matrix once - I've yet to meet a major one that skips it.
Full traffic is not always what you want; doubling the load on day one rarely is. The spec has a fraction field for that - numerator over denominator (default 100) - so you can start by mirroring 1% and turn the dial as confidence grows.
What it's for
The rewrite scenario is the obvious one: run v2 in the shadow for a week, compare its error rates and latency histograms against v1 on identical input, and the argument about readiness settles itself. The same trick warms caches before a cutover - the shadow instance has hot caches on switch day because it's been serving phantom traffic all week. And for the bug that only reproduces in prod, a mirror to a debug deployment with verbose logging gets you the evidence without touching the pod that's actually serving users.
Where it bites
The trap that matters is side effects. A mirrored request looks exactly like a real one to the service receiving it. If your shadow v2 writes to the production database, sends emails or charges cards, congratulations - you've built a duplication engine. The shadow needs its own isolated database, stubbed external calls, or a strictly read-only path. I'd treat this as a hard gate: no mirror until someone has written down where every write in the shadow path lands.
Two smaller ones. The gateway pays for the cloning - at high traffic, mirroring doubles the data plane's work, so watch CPU on the gateway pods, not just the backends. And the mirrored request keeps the original Host header; if the shadow routes by hostname, you'll need a RequestHeaderModifier in the chain before things line up.
The last thing isn't a trap so much as a prerequisite: the gateway throws the shadow's responses away, so metrics, traces and logs are the only window into whether v2 actually behaved. Mirroring without observability on the shadow is just heating the datacenter.
Links
- Ilia