On Tuesday, September 1, we raised one number in the checkout-api ConfigMap. upstream_timeout went from 2s to 5s, because a payment provider's p99 had crept up to 3.1 seconds. Twelve replicas. The service has had hot reload since 2024, so nobody restarted anything. Argo CD showed Synced, and the ConfigMap in the cluster said 5s.
By Wednesday morning timeout errors were down from about 410 an hour to about 240. I took that as the provider still being slow and moved on.
On Friday a node pool upgrade drained every node the service ran on. All twelve pods came back elsewhere, and timeout errors fell to 9 an hour. Three days of a config change that had "rolled out" and mostly hadn't.
Five pods had crashed, and those five got the new value
On Monday I split the error graph per pod. From Tuesday night on, seven pods produced every timeout and five produced none. Those five all had a restart count of 1 or 2, OOMKilled on Tuesday night by a memory limit set too tight months earlier. The seven that never restarted had been reading 2s all along.
Then I opened the chart. It mounted the ConfigMap with subPath: app.yaml at /etc/checkout/app.yaml, because the image ships other files in /etc/checkout that a directory mount would hide. The docs cover this in one note: "A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates." I'd read that page before. It hadn't stuck.
Links
What the kubelet writes into a ConfigMap volume
I reproduced it in staging: a throwaway pod mounting the same ConfigMap as a directory, next to a checkout-api pod with the subPath mount. I bumped the timeout from 5s to 7s. 71 seconds later the probe pod saw 7s, and the app pod still read 5s.
$ kubectl -n stg exec cfg-probe -- ls -la /cfg
total 12
drwxrwxrwx 3 root root 4096 Sep 7 10:14 .
drwxr-xr-x 1 root root 4096 Sep 7 10:12 ..
drwxr-xr-x 2 root root 4096 Sep 7 10:14 ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root 32 Sep 7 10:14 ..data -> ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root 15 Sep 7 10:12 app.yaml -> ..data/app.yamlNothing in that directory is a plain file. app.yaml links to ..data/app.yaml, and ..data links to a timestamped directory holding the real bytes. On an update, the kubelet's AtomicWriter writes the new payload into a fresh timestamped directory, points a ..data_tmp symlink at it and renames that over ..data, so a reader never catches half a file. Then the old directory gets deleted.
A subPath mount never looks at ..data again. At container start the kubelet runs the path through filepath.EvalSymlinks, lands on the real file inside the timestamped directory and bind-mounts that one file into the container. The swap moves ..data later, and the bind mount keeps holding the old file - deleted from the volume, still readable through the mount. A container restart makes the kubelet resolve the path again, see a different inode and remount (PR #89629, in since v1.19), which is why our OOMKilled pods were the lucky ones. Env vars from a ConfigMap are also read once, at start.
Our 71 seconds fit the documented formula: kubelet sync period plus cache propagation delay. Defaults are Watch for configMapAndSecretChangeDetectionStrategy and 1m for syncFrequency, and the pod worker requeues each pod with a 0.5 jitter factor, so a running pod gets re-synced roughly every 60 to 90 seconds.
Links
A CVE lived in the same resolve-then-mount step
I only found this one in the issue tracker while reading up on subpath_linux.go. CVE-2021-25741 was a symlink exchange in that code path: a container's subPath mounts could reach files outside the volume, host filesystem included, which gave hostPath-like access on clusters that had banned hostPath. It was rated High and fixed in v1.22.2, v1.21.5, v1.20.11 and v1.19.15, the second subPath symlink CVE after CVE-2017-1002101. Today's kubelet opens the subpath safely and bind-mounts /proc/<kubelet pid>/fd/<fd> instead of a path the container could swap.
Links
What we changed
checkout-api lost its subPath. The ConfigMap now mounts as a directory at /etc/checkout/conf.d/, with the config flag pointing there. Its hot reload is viper's WatchConfig, which watches the parent directory and re-checks the resolved symlink target on every event (the source comment names "k8s ConfigMap replacement"). Three edits in staging gave three reloads.
The ledger service was worse off. Its hand-rolled reload put an fsnotify watch on the file path, and inotify follows symlinks, so the watch sat on the real file inside the timestamped directory. The first swap deleted that directory and the watch with it, and no second update ever arrived. We moved the watch to the directory and re-read on any event touching ..data, which is what the AtomicWriter comment suggests.
Services with no reload got a checksum/config annotation on the pod template, so a ConfigMap edit changes the template and the Deployment rolls normally. Our two third-party charts went under Stakater's Reloader (10.4k stars, v1.4.22 on September 9) with reloader.stakater.com/auto set to "true".
ConfigMaps from Kustomize's configMapGenerator already carry a content-hash suffix, so those got immutable: true. The docs sell it on performance: watches on immutable objects get closed, which counts at tens of thousands of ConfigMap-to-pod mounts (more on what watches cost the apiserver). We're nowhere near that. I wanted the next in-place edit to fail at apply time, loudly.
The honest part
Hot reload through a directory mount skips the rollout. When I pushed a bad 50ms timeout on purpose to staging, all four replicas were serving with it 80 seconds later. A checksum rollout would have gone pod by pod under maxUnavailable. That route is slower and noisier, and I trust it more for anything that can break request handling.
Links
Where it landed
On September 10 we changed upstream_timeout again. The per-pod error graph moved as one line this time, and all twelve pods logged the reload within 88 seconds of the Argo CD sync.
Questions? Feedback? Reply to this email. I actually read them.
Ilia