Issue #030 - Mount storm: 20,200 syscalls to start 100 pods
User namespaces made containers safer and rewrote the arithmetic of startup, and the global lock underneath is a pattern you have met before
The formula is the whole story, so here it is before anything else:
100 containers x 2 x (1 + 50 + 50) = 20,200 mount operationsThat's what Netflix measured on the startup path of a hundred containers with fifty image layers each, after moving from Docker to containerd. Twenty thousand mount syscalls, every one of them taking a global kernel lock, to launch a hundred pods that between them don't mount a single volume.
Nobody wrote that. It emerged from three reasonable decisions - use user namespaces, use overlayfs, use containerd - each defensible, and the multiplication between them nowhere in any of the three designs. That combination is what makes it worth an issue: not a bug in any component, an interaction that only exists at density and only on a node packed the way real nodes get packed.
🏗️ Architectural Pattern: where 20,200 mounts come from
The security upgrade that changed the arithmetic
Old model: a shared host UID range for every container. When a layer was unpacked, file ownership got shifted once, at untar time, on disk. Cheap, done, never thought about again. Also weak - a container escape landed you in a UID space shared with every other container on the box.
New model: each container gets its own host UID range. A break-out from one lands nowhere useful, because the identity space doesn't overlap with anything. This is a genuine improvement and it's the direction everything is going.
The kernel implements it with idmapped mounts. Rather than rewriting ownership on disk, you create a mount of the layer that presents a shifted view of ownership, per container. Per container is the load-bearing phrase. Two containers sharing an image layer can't share its idmapped view, because the whole point is that their UID ranges differ.
So for each layer, per container, the runtime performs the modern mount dance:
open_tree() # get a detached handle on the layer directory
mount_setattr() # attach the UID/GID mapping to that handle
move_mount() # put it where the overlay expects itFifty layers, fifty of those. Then one overlayfs mount stacking them. Then, on teardown, fifty unmounts.
And containerd walks that path twice. Once to read user information out of the image so it knows what UID to run as, once to build the actual rootfs. Which gives 2 x (1 + 50 + 50) = 202 mount operations per container, and a hundred containers coming up together makes 20,200.
The lock underneath
Mount operations aren't independent. Linux keeps the mount namespace in a global structure guarded by global locks, and every mount, unmount, and move_mount takes them. A hundred containers starting in parallel don't get a hundred parallel mount paths. They get one, with ninety-nine waiting.
The symptoms Netflix saw are the recognisable shape of this. Reading /proc/mounts taking thirty seconds and more, because generating that file walks the mount table while everyone else is mutating it. systemd falling behind processing mount events. kubelet timing out on containerd, because containerd was blocked in the kernel rather than doing anything wrong.
None of those look like "too many mounts" in a dashboard. They look like a control plane that has gone soft, and the natural response - restart kubelet, drain the node - makes it worse, because a drain is a large batch of unmounts followed by a large batch of mounts somewhere else.
Links
🆚 The Showdown: the hardware fix vs the arithmetic fix
Two ways out, and the comparison is more interesting than either one alone, because they're the same two options you always get with lock contention.
Make the lock cheaper. Global lock throughput depends on how fast cores can pass a cache line between them, and on a modern server that's a function of the interconnect. A monolithic mesh die and a chiplet design with cross-die hops behave very differently under heavy contention on one line, and the gap is large enough that Netflix routed demanding workloads toward the architectures that degrade more gracefully.
It works. It also has an unsatisfying property: nothing got faster, the same 20,200 operations still happen, you just bought hardware that suffers less. The bottleneck is unchanged and the next density increase finds it again.
Do less work. The upstream direction is to stop mounting per layer. Map a common parent directory once, with the container's mapping, and let overlayfs address all the layers underneath it through that single idmapped view. One mount per container instead of one per layer, so the per-container cost drops from O(layers) to O(1), and the hundred-container number drops by roughly two orders of magnitude.
That's the real fix, and it's the one that keeps working when someone doubles the pod density or ships an image with ninety layers.
The comparison generalises past mounts, which is why I keep coming back to it. Kernel global lock plus linear work per unit of workload gives you a scaling limit that's completely invisible until you cross it, and then arrives all at once. Postgres had this with ProcArrayLock and connection counts, fixed in 14 by making snapshots scale rather than by asking people to buy better CPUs. The apiserver has a version of it in watch fan-out under write storms. The pattern is worth carrying around: when you profile a scale problem and find one lock, ask what's linear in front of it before you go shopping.
And a practical corollary. Microbenchmarks won't show you any of this. One container starting on a laptop performs the same 202 mounts and never contends, because there's nobody to contend with. Density limits have to be tested at density, on the CPU topology you actually run.
Links
👮 The Policy: layer count is a startup-latency metric
The reader-actionable part is small and nobody does it: count the layers in your images.
crane config <image> | jq '.rootfs.diff_ids | length'Or across everything running:
kubectl get pods -A -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' \
| sort -uthen run the first command over that list. Most people are surprised twice - once by the maximum, once by how many images sit above forty when the base image alone accounts for a dozen.
Layer count has been treated as an image-size concern for a decade, and on that metric it barely matters, because layers dedupe and registries cache. On mount cost it's linear and it doesn't dedupe, because idmapped views are per container by construction. That reframes the usual image-hygiene advice: multi-stage builds, collapsing RUN chains, and distroless bases aren't only about pull time and CVE surface. They are also about how many times the kernel takes a global lock when your node reboots.
Worth putting a check in CI. A build that adds five layers to a base image is fine; one that ships sixty because every apt-get got its own line is a node-density problem waiting for a busy Monday.
The honest part
This is a density problem and most clusters aren't dense. At thirty pods a node with twenty-layer images you're looking at roughly 1,200 mount operations spread over a rolling start, and you'll never see it. The teams that hit this are running a hundred-plus pods per node, or restarting a whole node's worth of workloads at once, or both.
The other honest bit: you probably can't apply the real fix yourself. Whether the per-layer dance collapses into one mount is decided by your container runtime and kernel version, not by anything in your manifests. What you control is the multiplier. Fewer layers, and not restarting every pod on a node simultaneously when a rolling drain would do.
If you want to know whether it's you, mountsnoop from bcc-tools on a node during a scale-up will tell you in a minute. Count the syscalls and compare with the formula. If the number matches, you now know exactly which term to attack.
Links
The class of bug this belongs to
What I like about this one is that no individual decision was wrong. User namespaces per container is correct. Idmapped mounts are the right implementation. Overlayfs stacking layers is how images work. Containerd reading image config before building a rootfs is reasonable. Multiply them and you get twenty thousand syscalls through a single lock.
That's most interesting infrastructure failure, in my experience. Not a bad component, four good ones whose costs compose in a direction nobody was measuring, showing up only past a density threshold that the people who designed each piece never tested at.
I went and counted the layers in our own images after reading the Netflix writeup, expecting to feel smug. Our worst one is at forty-three. It's a Python service, and thirty of those layers are a base image that somebody assembled carefully, one dependency per layer, to get good cache behaviour in CI. Which was a completely sound decision, made for a completely different metric.
Questions? Feedback? Reply to this email. I actually read them.
Ilia