Nobody touched the production Redis namespace that week. That was the whole problem with the postmortem: every line of the incident had a completely different name in it, valkey-test, and it still ended with resources gone from redis-system. Four unrelated decisions, each one reasonable on its own, lined up into a chain that let a Kustomization adopt a namespace it had never been told to manage, then delete it again on schedule, exactly as configured.
Nothing here is a Flux bug. Every step is the tool doing precisely what its config says. That's what makes it worth walking through slowly: four defaults compounding, not a single obvious mistake.
The four ingredients
The production Redis Kustomization watched path: ./kubernetes/, not ./kubernetes/production/redis-system/ - the whole tree above it, and not the directory that actually belonged to it. Flux Kustomization doesn't ask which manifests under that path belong to it, either. It takes all of them recursively, by default. Then there's targetNamespace, which doesn't filter what Flux picks up - it force-rewrites metadata.namespace on every namespaced resource it finds, regardless of what namespace the manifest author actually wrote. And prune: true, turned on for exactly the reason you'd turn it on: keep the cluster from accumulating dead resources nobody remembers creating.
Individually these are all sane defaults. path has to point somewhere. Recursion is what makes a directory of manifests useful instead of requiring one Kustomization per file. targetNamespace exists so you can reuse the same manifests across environments. prune is the thing that makes GitOps actually mean "the cluster matches Git," instead of "the cluster is a superset of Git that only grows."
How the adoption actually happened
Someone added a kubernetes/test/ directory for a Valkey experiment, unrelated to the Redis rollout, just organized under the same parent tree. The Redis Kustomization's next reconciliation walked ./kubernetes/, found the new directory, and pulled every manifest in it into its own managed set, including a namespace manifest for valkey-test.
targetNamespace: redis-system then did exactly what it's documented to do. For cluster-scoped resources like the Namespace object itself, the field is a no-op: Flux just created valkey-test as specified, a stray but harmless namespace. For anything namespaced sitting in that same test directory (a Deployment, a Secret), targetNamespace would have silently rewritten its namespace from valkey-test to redis-system, moving a test workload's manifests into the production namespace without anyone editing a single YAML file to say so.
Weeks later, someone cleaned up by renaming kubernetes/test/ to kubernetes-test/, moving it outside the watched path, which reads as the responsible thing to do. On the next reconciliation, Flux compared its desired state (Git, where the adopted resources no longer appeared under the watched path) against actual state (the cluster, where they still existed, because Flux had created them). prune: true did what it's for: deleted everything it had adopted and could no longer see in Git. Namespace/valkey-test, gone. Anything else it had silently claimed, gone with it.
Links
The version where this isn't recoverable
What actually happened was close to the safe end of this failure mode: Flux deleted only what it had wrongly created itself, and the incident resolved as a strange namespace appearing and disappearing.
The version that doesn't resolve cleanly needs one more coincidence: a filename collision. If the wandering test directory had contained a secret.yaml that happened to share a name with a real production secret already managed under redis-system, one of two things happens instead. Either Flux's apply overwrites the real secret's contents with the test manifest's, silently, on the next reconcile, with no alert. Or, worse, the later prune reads the production secret as something it adopted and no longer sees in Git, and deletes the real one.
prune: true doesn't know the difference between a resource it should own and a resource it accidentally started owning. It only knows what's in Git and what's in the cluster under its watch. If the path is wide enough to accidentally include something it shouldn't, prune becomes, in the plainest sense, a weapon pointed at your own cluster by a rule that's doing exactly what it was told.
What actually closes the gap
Never point a Kustomization's
pathat a parent directory that might one day contain something unrelated../kubernetes/is the mistake,./kubernetes/production/redis-system/is the fix: the deepest, most specific directory that contains only that Kustomization's own manifests and nothing else, ever.If a shared parent path is genuinely unavoidable, a
.fluxignorefile at that path (same syntax as.gitignore) excludes subtrees liketest/,staging/, or*-test/from being swept up at all.targetNamespaceis a global override with no per-resource awareness, so stop reaching for it by default. Akustomization.yamlwith an explicitnamespace:field in the directory itself gives you the same result with none of the blast radius, because it only ever touches manifests that are already scoped to that directory.spec.wait: truemakes Flux wait for resources to report ready before considering the reconcile complete, which surfaces adoption-shaped surprises sooner, before they've had a chance to sit quietly for weeks.A single YAML change won't close this gap on its own. Treat "do any two Kustomizations' paths overlap" as a question worth asking on a recurring basis, the same way you'd audit IAM policies for overlap: the failure mode here is two good configs that were never checked against each other, not a bad config in isolation.
Links
Questions? Feedback? Reply to this email. I actually read them.
- Ilia