Podo Stack

Bitnami, one year later: hardened base images went free

Ilia Gusev — Fri, 18 Sep 2026 14:03:50 GMT

This week I ran helm show values on the Apache Airflow chart, version 1.22.0, released in June. The bundled PostgreSQL points at bitnamilegacy/postgresql, tag 16.1.0-debian-11-r15. The image labels say it was built on November 30, 2023, on Debian 11, and Debian 11 left LTS on August 31.

That image is there on purpose. A year ago Bitnami shut most of its free catalog and said running it cost too much. Eleven weeks later Docker gave its hardened images away under Apache 2.0, and by August 2026 a startup that matched that price had announced its shutdown.

September 29, 2025: the tags disappeared

Broadcom posted the notice on July 16, 2025, in bitnami/containers issue #83267. On August 28 Bitnami stopped publishing Debian-based images and OCI charts to Docker Hub, then ran brownouts through September.

The cut came on September 29. That morning someone filed an issue because docker.io/bitnami/kafka had no tags at all. The maintainer replied that Kafka wasn't in the trial set: about 40 images and 10 charts stayed free, latest tag only, for development. Versioned tags were frozen in docker.io/bitnamilegacy. Production moved to Bitnami Secure Images, rebuilt on VMware's Photon OS, with no public price; press coverage put it at $50,000 a year and up.

The reasoning came in a Broadcom community post on August 18: "Bitnami has been the Jenkins of the internet for many years, but this has become unsustainable. Operating a build pipeline and OCI registry for the general public is very expensive."

AWS dropped its 317 Bitnami repositories from ECR Public on June 10, 2026. Today docker.io/bitnami/postgresql has no version tags, only latest, which is PostgreSQL 18.6 on Photon OS 5.0:

$ docker pull bitnami/postgresql:16.4.0
Error response from daemon: manifest for bitnami/postgresql:16.4.0 not found: manifest unknown: manifest unknown

December 17, 2025: Docker gave its images away

Docker launched Docker Hardened Images on May 19, 2025, as a paid product with a 7-day patch window for critical and high CVEs. When Bitnami users went shopping that August, Docker called DHI "affordable and transparent", which still meant a trial and a sales conversation.

On December 17 more than 1,000 images and Helm charts went free under Apache 2.0, served from dhi.io. A Docker account is the entry ticket; an anonymous token request to dhi.io comes back Unauthorized. Unlike Bitnami's leftovers, the free tier keeps versioned tags such as python:3.13. Select and Enterprise hold the SLA and the FIPS and STIG variants, and Enterprise alone gets five years of Extended Lifecycle Support past upstream end of life. The public definitions repo held 595 image definitions and 96 charts when I counted this week.

One developer quoted by DevClass wasn't sold: "It's free for now, just like registries were 'free' and docker desktop was free... until they weren't."

2026: everyone priced against zero

Chainguard had the most to lose and moved the least. Its free images at cgr.dev/chainguard need no login but ship as latest and latest-dev only, a set it trimmed in November 2024. Anonymously, python:latest resolves for me and python:3.12 returns 404. In December CEO Dan Lorenc told TechTarget he'd be "pretty cautious" about Docker's offer. On March 17, 2026, Chainguard launched Catalog Starter: five images of your choice, all supported versions, production use allowed, no CVE SLA, corporate email required.

Red Hat followed on May 12 with Red Hat Hardened Images, a no-cost catalog of more than 45 images.

Minimus went furthest. On June 23 it opened its whole catalog, every major and minor version, no registration. By late August its homepage said the reg.mini.dev registry switches off on October 22. Echo bought the technology on August 27 and told TechTarget it would keep the platform running. Chris Hughes of Noma Security told the same outlet: "Everyone kind of raced to 'out-free' each other."

What "near-zero CVEs" counts

Every catalog above advertises near-zero CVEs, so I went looking for the denominator. Docker's number is what a scanner reports after applying Docker's own VEX statements. A CVE Docker judges unexploitable gets not_affected with a justification, and a backported fix without a version bump gets not_affected plus inline_mitigations_already_exist. Docker's docs warn that scanners without VEX support report more. The Grype example in those same docs, VEX applied, still shows a High perl CVE marked won't fix.

Chainguard publishes its triage as security advisories in wolfi-dev/advisories and secdb feeds that scanners consume. Its docs say plainly that a CVE count depends on build age, and show an October 2024 jre digest scanning dirty. The paid SLA clock starts once a qualifying patch exists: seven days for critical, fourteen for everything else.

The honest reading of zero is a fresh build scanned by a tool that accepts the vendor's triage. Pin that digest for six months and you run a six-month-old image with the same badge.

What teams did with their charts

On September 18, 2025, mid-brownout, Airflow merged "Temporary fix to Bitnami psql chart licensing issues", which swapped in the official postgres:13 image, a major-version downgrade from 16. In January 2026 a follow-up put back the exact Bitnami image, now from the archive: "It's likely users of the provided postgres moved to this image anyways." Mastodon pointed its Bitnami subchart images at bitnamilegacy in October 2025. Superset did the same on May 4, 2026, "until we can switch over to Valkey and vanilla Postgres."

GitLab split it. The company's infrastructure team parked its Redis caches on bitnamilegacy/redis as a short-term fix, while the product chart removed bundled Bitnami PostgreSQL and Redis in GitLab 19.0 on May 21, 2026, "with no replacement", and the migration guide points at CloudNativePG and Valkey. Langfuse's chart 2.0.0, out August 18, replaced Bitnami Redis with the official Valkey chart and Bitnami Postgres with groundhog2k/postgres. The Valkey issue covered where that chart came from.

The advice from both ends of the removal was mirroring. A Bitnami maintainer told one user to copy an image "to your own registry before the final removal." AWS, retiring its own copies, wrote: "Never point production workloads directly at a public registry you don't own." A mirror with digest pins trades a tag that vanishes for a digest that never changes, and someone still has to move the pin.

The recipe was never the expensive part

Bitnami's Dockerfiles still sit in bitnami/containers under Apache 2.0, with commits from this week. Anyone can build them. Broadcom's complaint was about running "a build pipeline and OCI registry for the general public", and the vendors doing that for free in 2026 have something else to sell next to it. Docker has subscriptions and Red Hat has RHEL; Minimus had only the images.

Until somebody opens the next PR, Airflow's chart will keep pulling a Postgres build from November 2023.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

subPath: the ConfigMap update 7 of our 12 pods never saw

Ilia Gusev — Wed, 16 Sep 2026 14:03:40 GMT

On Tuesday, September 1, we raised one number in the checkout-api ConfigMap. upstream_timeout went from 2s to 5s, because a payment provider's p99 had crept up to 3.1 seconds. Twelve replicas. The service has had hot reload since 2024, so nobody restarted anything. Argo CD showed Synced, and the ConfigMap in the cluster said 5s.

By Wednesday morning timeout errors were down from about 410 an hour to about 240. I took that as the provider still being slow and moved on.

On Friday a node pool upgrade drained every node the service ran on. All twelve pods came back elsewhere, and timeout errors fell to 9 an hour. Three days of a config change that had "rolled out" and mostly hadn't.

Five pods had crashed, and those five got the new value

On Monday I split the error graph per pod. From Tuesday night on, seven pods produced every timeout and five produced none. Those five all had a restart count of 1 or 2, OOMKilled on Tuesday night by a memory limit set too tight months earlier. The seven that never restarted had been reading 2s all along.

Then I opened the chart. It mounted the ConfigMap with subPath: app.yaml at /etc/checkout/app.yaml, because the image ships other files in /etc/checkout that a directory mount would hide. The docs cover this in one note: "A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates." I'd read that page before. It hadn't stuck.

What the kubelet writes into a ConfigMap volume

I reproduced it in staging: a throwaway pod mounting the same ConfigMap as a directory, next to a checkout-api pod with the subPath mount. I bumped the timeout from 5s to 7s. 71 seconds later the probe pod saw 7s, and the app pod still read 5s.

$ kubectl -n stg exec cfg-probe -- ls -la /cfg
total 12
drwxrwxrwx 3 root root 4096 Sep  7 10:14 .
drwxr-xr-x 1 root root 4096 Sep  7 10:12 ..
drwxr-xr-x 2 root root 4096 Sep  7 10:14 ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root   32 Sep  7 10:14 ..data -> ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root   15 Sep  7 10:12 app.yaml -> ..data/app.yaml

Nothing in that directory is a plain file. app.yaml links to ..data/app.yaml, and ..data links to a timestamped directory holding the real bytes. On an update, the kubelet's AtomicWriter writes the new payload into a fresh timestamped directory, points a ..data_tmp symlink at it and renames that over ..data, so a reader never catches half a file. Then the old directory gets deleted.

A subPath mount never looks at ..data again. At container start the kubelet runs the path through filepath.EvalSymlinks, lands on the real file inside the timestamped directory and bind-mounts that one file into the container. The swap moves ..data later, and the bind mount keeps holding the old file - deleted from the volume, still readable through the mount. A container restart makes the kubelet resolve the path again, see a different inode and remount (PR #89629, in since v1.19), which is why our OOMKilled pods were the lucky ones. Env vars from a ConfigMap are also read once, at start.

Our 71 seconds fit the documented formula: kubelet sync period plus cache propagation delay. Defaults are Watch for configMapAndSecretChangeDetectionStrategy and 1m for syncFrequency, and the pod worker requeues each pod with a 0.5 jitter factor, so a running pod gets re-synced roughly every 60 to 90 seconds.

A CVE lived in the same resolve-then-mount step

I only found this one in the issue tracker while reading up on subpath_linux.go. CVE-2021-25741 was a symlink exchange in that code path: a container's subPath mounts could reach files outside the volume, host filesystem included, which gave hostPath-like access on clusters that had banned hostPath. It was rated High and fixed in v1.22.2, v1.21.5, v1.20.11 and v1.19.15, the second subPath symlink CVE after CVE-2017-1002101. Today's kubelet opens the subpath safely and bind-mounts /proc//fd/ instead of a path the container could swap.

What we changed

checkout-api lost its subPath. The ConfigMap now mounts as a directory at /etc/checkout/conf.d/, with the config flag pointing there. Its hot reload is viper's WatchConfig, which watches the parent directory and re-checks the resolved symlink target on every event (the source comment names "k8s ConfigMap replacement"). Three edits in staging gave three reloads.

The ledger service was worse off. Its hand-rolled reload put an fsnotify watch on the file path, and inotify follows symlinks, so the watch sat on the real file inside the timestamped directory. The first swap deleted that directory and the watch with it, and no second update ever arrived. We moved the watch to the directory and re-read on any event touching ..data, which is what the AtomicWriter comment suggests.

Services with no reload got a checksum/config annotation on the pod template, so a ConfigMap edit changes the template and the Deployment rolls normally. Our two third-party charts went under Stakater's Reloader (10.4k stars, v1.4.22 on September 9) with reloader.stakater.com/auto set to "true".

ConfigMaps from Kustomize's configMapGenerator already carry a content-hash suffix, so those got immutable: true. The docs sell it on performance: watches on immutable objects get closed, which counts at tens of thousands of ConfigMap-to-pod mounts (more on what watches cost the apiserver). We're nowhere near that. I wanted the next in-place edit to fail at apply time, loudly.

The honest part

Hot reload through a directory mount skips the rollout. When I pushed a bad 50ms timeout on purpose to staging, all four replicas were serving with it 80 seconds later. A checksum rollout would have gone pod by pod under maxUnavailable. That route is slower and noisier, and I trust it more for anything that can break request handling.

Where it landed

On September 10 we changed upstream_timeout again. The per-pod error graph moved as one line this time, and all twelve pods logged the reload within 88 seconds of the Argo CD sync.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

Issue #035 - HolmesGPT, Job retries, and the IfNotPresent hole from 2015

Ilia Gusev — Tue, 15 Sep 2026 14:03:08 GMT

Capsule's API description says a tenant quota's hard limit "is never crossed." The controller behind it sums usage on a reconcile loop and rewrites each namespace's limit to whatever headroom is left, so between two passes every namespace sees the same leftover. I only found that because I stopped at the sentence and opened the code.

It happened again in all four blocks this week. Grab bag, round three.

🚀 Sandbox Watch: HolmesGPT

What it is

An open-source agent for incident investigation. You ask "why is ledger crashlooping?", it queries your cluster and observability stack, then writes up a root cause. Robusta started it in May 2024 and donated it to the CNCF, where the Sandbox vote passed on October 8, 2025. The repo moved from robusta-dev to its own HolmesGPT org. Apache 2.0, about 3,350 stars, release 0.41.0 on September 8.

How it works

An agentic loop, where the model reads each tool result before choosing its next call. Built-in toolsets reach Kubernetes, Prometheus, Loki, Tempo, Datadog, ArgoCD and dozens more, plus MCP servers. Runbooks are now called skills, markdown procedures synced from git. Models go through LiteLLM, with OpenAI as the default and Anthropic, Bedrock or Ollama a flag away. It ships as a CLI, a Helm-deployed HTTP API, a k9s plugin, or a Slack bot via Robusta.

The honest part

"By design, HolmesGPT has read-only access," says the README. I went looking for where that gets enforced. The Kubernetes toolset skips secrets, and the Helm chart's ClusterRole grants none. A bash toolset ships enabled too, though, with "kubectl get" in its core allowlist and an empty default deny list. On the CLI it runs under your kubeconfig, so a secret read passes whenever your RBAC permits it. Anything off the list stops at a "Do you want to proceed?" prompt, which --bash-always-allow removes. Since February a Remediation MCP server can restart, scale, drain and patch, each change human-approved. That's the policy question from issue #034, inside an SRE tool.

Every log line the loop pulls lands in the model's context, off your network with a hosted provider. Ollama support is labeled experimental, with tool calling that "may produce inconsistent results."

Accuracy, from the project's own August 5 benchmark: seven models passed between 71% and 94% of 63 runs, and five scored 50% on tests tagged "hard". A wrong answer prints under the same "AI:" header as a right one.

Where I'd use it

k8sgpt, Sandbox since December 2023, finds problems with Go analyzers and calls an LLM only for --explain. HolmesGPT lets the model choose what to query, so it reaches further and guesses more. I'd take it for the first ten minutes at 3 a.m., read its conclusion as a list of places to look, and keep --bash-always-allow away from any admin kubeconfig.

💎 The Hidden Gem: backoffLimitPerIndex

What it is

A retry budget per index for Indexed Jobs. backoffLimitPerIndex and maxFailedIndexes went GA in Kubernetes 1.33 and have been on by default since 1.29. Around them the Job API kept growing: podFailurePolicy reached GA in 1.31, successPolicy in 1.33, podReplacementPolicy in 1.34, managedBy in 1.35. The retry knob most people remember is still the original one, backoffLimit, default 6.

One budget for a thousand shards

Picture a 1000-shard Indexed Job with that setting. The controller compares the 6 against status.failed for the whole Job, a counter that doesn't reset when other shards succeed, and fails the Job on the seventh failed pod from any index. Three crashes on shard 7 plus four pods caught in a node drain will do it. Running pods get deleted, unstarted indexes never get a pod, and nothing retries the Job afterwards.

Drains count immediately. Without a pod failure policy, a pod is treated as failed the moment it gets a deletionTimestamp, even if it would have exited 0 inside its grace period.

The docs say retry backoff (10s, doubling) caps at six minutes. I checked pkg/controller/job, and the cap constant has been 10 * time.Minute since 1.28.

What per-index changes

Each index gets its own failure counter and its own backoff clock. A shard that burns through its budget lands in status.failedIndexes while the other 999 keep going. Once failed indexes go past maxFailedIndexes, the controller terminates the Job. Under that cap it still ends Failed, reason FailedIndexes, but with 997 shards done and three index numbers to rerun.

An Ignore rule on the DisruptionTarget pod condition keeps evictions and preemptions off the count, and FailIndex gives up on an index at once on an exit code reserved for bad input.

apiVersion: batch/v1
kind: Job
metadata:
  name: reindex-shards
spec:
  completionMode: Indexed
  completions: 1000
  parallelism: 50
  # no backoffLimit here: with backoffLimitPerIndex set it defaults to 2147483647
  backoffLimitPerIndex: 2      # third failed pod marks this index failed
  maxFailedIndexes: 20         # 21st failed index terminates the Job
  podFailurePolicy:
    rules:
    - action: Ignore           # drain, preemption, taint eviction, node pressure
      onPodConditions:
      - type: DisruptionTarget
    - action: FailIndex        # bad input for this shard, retrying won't help
      onExitCodes:
        containerName: worker
        operator: In
        values: [42]
  template:
    spec:
      restartPolicy: Never     # required by both backoffLimitPerIndex and podFailurePolicy
      containers:
      - name: worker
        image: registry.example.com/reindex:1.8.2
        args: ["--shard=$(JOB_COMPLETION_INDEX)"]

Where it still bites

Leftover backoffLimit in the manifest still wins. The controller checks the Job-wide limit first, so a chart that templates backoffLimit: 6 onto every Job brings the shared budget back.

Ignoring DisruptionTarget also swallows kubelet node-pressure evictions. If a shard pushes its node into memory pressure, it gets rescheduled with no retry limit unless activeDeadlineSeconds is set. An OOM kill against the container's own memory limit still counts.

kubectl describe job prints Completed Indexes and nothing for failed ones; those live only in status.failedIndexes.

Version lag is small. The oldest EKS release still in extended support is 1.31, and upstream every field above is on by default there. managedBy is the exception, alpha and off until 1.32.

⚔️ The Showdown: Only one of these tenancy tools changes your namespace count

The setup

I went to pull the Hierarchical Namespace Controller into this comparison and found a read-only repo under kubernetes-retired. SIG Auth voted in February 2025 to archive it "due to a lack of maintainers and adopters".

How each one draws the line

Accurate copies objects down a tree. Annotate a Secret or Role with accurate.cybozu.com/propagate and it lands in the sub-namespaces below. Budgets are out of scope. The sample config even lists ResourceQuota as a propagatable kind, and a copied quota gives each child the full limit. v1.9.0 in June was mostly Renovate bumps. 79 stars.

Capsule is the governance layer: a Tenant owns a capped set of namespaces and pins allowed registries and storage classes. It has sat in CNCF Sandbox since December 2022 and has 2,175 stars; v0.14.5 shipped last Friday. The tenant quota's API description promises the hard limit "is never crossed", so I read the controller. The legacy spec.resourceQuotas is a reconcile loop that sums usage and rewrites each namespace's hard to the leftover headroom. Between passes, every namespace sees the same leftover. v0.14.0 (August 21) added GlobalResourceQuota, which reserves usage atomically at admission, and the old quota system is now deprecated.

With vCluster, the tenant stops being a namespace. Each one gets its own API server, OSS under Apache 2.0. Embedded etcd and Private Nodes need the Free tier, which validates its license against vCluster Platform. v0.37.1 landed yesterday, 11,298 stars.

The #023 cost angle

In #023 the bill scaled with namespace count. Capsule and Accurate leave that count where it was, and Accurate inflates the object count on top: one Secret propagated over 40 sub-namespaces is 41 Secrets in every cluster-wide Secret informer.

vCluster moves the line. Tenant namespaces and CRDs stay in the virtual control plane; the syncer pushes pods, services, endpoints and PVCs, plus only the ConfigMaps and Secrets pods reference, into one host namespace per tenant.

The honest part: pods still run on host nodes, so pod-watching DaemonSets see every one of them. And the bill moves into control planes. The chart default asks for 200m CPU and 256Mi, but the sizing guide recommends 4 CPU and 8 GiB per replica, three replicas, for production. Fifty tenants at that profile is 600 CPU in limits.

My read

Trusted teams sharing Secrets and RBAC

Accurate. Opt-in copying, nothing more.

Internal platform with real per-team budgets

Capsule 0.14+, shared limits in GlobalResourceQuota.

Namespace count already hurting, or tenants need their own CRDs

vCluster, after pricing the control planes.

Still on HNC

Archived since April 2025. The archive thread points at Accurate.

👮 The Policy: IfNotPresent let pod B run an image it had no right to pull

Pod B never showed a credential

Pod A in team-a runs registry.example.com/private/app:1.4.2 with an imagePullSecret. Later the scheduler drops pod B from team-b onto the same node - same image, no imagePullSecrets, default IfNotPresent. The kubelet asks the runtime whether the image is on disk, hears yes, and logs "Container image ... already present on machine". No registry call, so no credential check. kubernetes/kubernetes#18787 reported this in December 2015 and closed in April 2025.

The archive note behind this block forced Always through a Kyverno ClusterPolicy and promised "no limitations." Neither half holds up in 2026.

What the fixes cost

AlwaysPullImages. An admission plugin, off by default, that sets Always on every Pod and rejects anything else. For an unchanged image the pull is a manifest check measured in kilobytes, per KEP-2535. The bill is availability: the kubelet re-checks on every container start, CrashLoopBackOff restarts included, so a registry outage stops pods whose bytes already sit on the node.

KEP-2535, inside the kubelet. The KubeletEnsureSecretPulledImages gate went alpha in v1.33 and flipped to beta, on by default, in v1.35. v1.37 still ships it as beta. Every kubelet pull leaves a record under /var/lib/kubelet/image_manager/ with the secret's namespace, name, UID and credential hash, kept across restarts. Pod B brings no matching credential, so the kubelet pulls for real and the private registry refuses it. A pod reusing a known secret skips the registry.

The policy kind is on its way out too: Kyverno 1.19 deprecated ClusterPolicy on August 20, and 1.20 removes it.

The manifests

For 1.35+ nodes whose kubelet config you own:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# NeverVerify | NeverVerifyPreloadedImages (default)
# NeverVerifyAllowlistedImages | AlwaysVerify
imagePullCredentialsVerificationPolicy: NeverVerifyAllowlistedImages
preloadedImagesVerificationAllowlist:
  - registry.k8s.io/*   # public only: allowlisted images skip verification entirely

For nodes you can't configure, or older than 1.35, force Always for the private prefix and nothing else. MutatingAdmissionPolicy went GA in v1.36, so there's no webhook to run:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
  name: private-images-always-pull
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE"]
      resources: ["pods"]
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >
        Object{
          spec: Object.spec{
            containers: object.spec.containers
              .filter(c, c.image.startsWith("registry.example.com/"))
              .map(c, Object.spec.containers{name: c.name, imagePullPolicy: "Always"})
          }
        }
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >
        Object{
          spec: Object.spec{
            initContainers: object.spec.?initContainers.orValue([])
              .filter(c, c.image.startsWith("registry.example.com/"))
              .map(c, Object.spec.initContainers{name: c.name, imagePullPolicy: "Always"})
          }
        }
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicyBinding
metadata:
  name: private-images-always-pull
spec:
  policyName: private-images-always-pull

I ran it against a v1.37.0 kube-apiserver: both private containers came back Always, and an nginx:1.27 sidecar kept IfNotPresent. Kyverno 1.18+ takes the same mutations block in a policies.kyverno.io/v1 MutatingPolicy.

What stays open

Under the default policy, any image the kubelet didn't pull itself counts as preloaded and stays open to every pod on the node. That includes crictl-based warmers like the DaemonSet in #020, and everything already on disk when the gate first turned on. An in-place upgrade from 1.34 leaves a node full of those. AlwaysVerify closes the gap and puts the registry back in front of them.

GKE's node system config has no field for this policy; EKS AL2023 nodes take raw KubeletConfiguration through nodeadm. Records don't expire either: revoke a registry token and pods referencing that same secret keep starting the cached image until image GC removes it.

Links

Issue #18787 went up on GitHub in December 2015 and was closed in April 2025. The default that shuts the gap shipped with v1.35 that December, ten years after the report. It covers images the kubelet pulled itself, so a crictl preloader like the one in #020 still leaves pod B a way in.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

The rolling update myth: your frontend is talking to next week's backend

Ilia Gusev — Fri, 11 Sep 2026 14:02:38 GMT

A user loads your app at 2:47pm. Their browser caches the JavaScript bundle - React Server Component payloads, API client code, the works. At 2:52pm, your rolling update finishes: new backend pods are live, old ones are gone. The user, still on the same tab, clicks something. Their cached frontend, built for deployment N, calls an API shaped for deployment N+1.

Sometimes that's a 500. Sometimes it's a silent RSC hydration failure. Other times it's a contract mismatch that only shows up as "weird" data three clicks later. Nobody on call connects it to the deploy, because the deploy finished five minutes ago and "succeeded."

Rolling updates are safe for the backend. They say nothing about whether the client talking to that backend is the version you think it is.

Why this is worse in full-stack frameworks

Any API can drift out of sync with an old client for a few seconds during a rollout - that's not new. What's new is how tightly coupled frontend and backend have become in frameworks like Next.js, Remix, SvelteKit, and Nuxt: shared types, server components whose payload shape is the contract, code that assumes the two sides were built together because, until a rolling update happens, they were.

And the window isn't seconds anymore. A user's session - and the JS bundle their browser is holding onto - can outlive your rolling update by minutes, hours, sometimes days, depending on caching and how long they leave the tab open. "The rollout only takes two minutes" doesn't help if the client that matters loaded ten minutes before it started.

What's actually going on: version skew

This has a name - version skew - and once you know to look for it, the fix is controlling which backend version a given client session actually talks to, for as long as that session lives, not shipping rollouts faster.

Vercel shipped this first, for Next.js specifically: Skew Protection. A deployment ID gets attached to the client, carried via a cookie (__vdpl), and every subsequent request from that session gets routed back to the exact deployment that originally served it - not whatever's currently live. The session stays pinned to one version until its own natural end, regardless of how many deploys happen underneath it.

Doing it on plain Kubernetes

Vercel's version is a managed platform feature. The concept ports to vanilla Kubernetes without needing that platform, using pieces you likely already have: a session cookie that carries a deployment identifier, Gateway API routing rules that read that cookie and send the request to the matching backend version, and a small state machine tracking each deployment as Active, Draining, or Expired so you know when it's finally safe to tear an old version down.

None of this replaces your rollout strategy - canary, blue/green, whatever you're already doing. Skew protection sits on top of it. The rollout question is "how do I move traffic to the new version safely." The skew question is a separate one: "once a specific user's session started on version N, does it stay on version N until that session naturally ends."

The trade-off that makes this non-trivial

Pinning sessions to versions means old versions have to keep running, potentially for as long as your longest-lived session - which means running N versions of your backend simultaneously instead of one, with memory, replica count, and infrastructure cost scaling with N. Observability gets messier too: every metric now needs a version label, or your dashboards quietly average together three different backends and tell you nothing useful.

It's also not a fit for everything. A clean backend API with no shared types and no client-side caching to speak of mostly doesn't have this problem in the first place - skew protection is solving a specific coupling, not a general Kubernetes deployment concern.

When to reach for this

Anywhere you don't control the client cache - mobile apps that don't force-refresh, CDN edge caching, long-lived SPA sessions - rolling updates alone aren't enough, and the fix looks more like blue/green scoped to the session than a faster rollout. It's the failure mode that falls squarely between the frontend team and the platform team, which is usually why it ships unfixed: each side assumes the other one owns it.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The scheduler that starts 7 of your 8 GPU pods, then bills you for all 8

Ilia Gusev — Wed, 09 Sep 2026 14:02:52 GMT

A distributed training job asks for eight GPU pods. Seven schedule within seconds. The eighth sits in Pending - one node short of capacity somewhere in the cluster - and stays there. Nothing crashes. Nothing errors. The training process on all seven running pods just hangs, because the all-reduce step that synchronizes gradients across the job is waiting for a peer that never showed up.

Seven GPUs, burning money, doing nothing, for however long it takes someone to notice and kill the job.

The default Kubernetes scheduler did exactly what it was built to do: schedule pods, one at a time, as capacity allows. That's precisely the problem.

Why this isn't a `podAffinity` problem

The instinct is to reach for podAffinity or topologySpreadConstraints to keep the job's pods close together. Neither one touches the actual failure. Both operate per-pod, evaluated independently as each pod is scheduled - they can influence where a pod lands, but nothing about them makes eight pods land together or not at all. The scheduler still admits each pod as its own decision, and a job that needs all eight can still end up with seven.

Distributed training, MPI jobs, Spark executors that all need to see each other before work starts - none of these have a "partial success" mode. A job that gets 7 of 8 pods isn't 87% done. It's 0% done, at full cost.

Gang scheduling: the actual fix

Gang scheduling flips the admission decision from per-pod to per-job: either every pod in the group gets scheduled together, or none of them do, and the ones that would've scheduled successfully get held back too. HPC batch schedulers have handled this exact problem for decades; Kubernetes borrowed the concept late, and it shows up now as "all-or-nothing" semantics in a handful of projects: Kueue, Volcano, Apache YuniKorn.

Kueue is the one worth knowing first: an official Kubernetes SIG-Scheduling project (kubernetes-sigs/kueue), not some third-party fork of the scheduler, currently at v0.13.4 and shipping regularly.

How Kueue actually enforces it

Kueue sits in front of the default scheduler as an admission layer, rather than replacing it. Jobs go into a LocalQueue, a namespace-scoped object your team submits work to, which references a ClusterQueue that owns the actual quota and fair-sharing rules across the cluster. Kueue holds a job back until it's confident the whole group can be admitted, then releases all of its pods to the underlying scheduler together.

The enforcement mechanism is waitForPodsReady: a cluster-wide, timeout-based check. Once Kueue admits a workload, it watches until every pod in it reports ready. If the timeout passes before that happens - because a node disappeared, or capacity got contended by something else - Kueue evicts the whole workload and requeues it, rather than leaving it half-running and silently burning resources. That's the mechanism that turns "7 of 8, hanging forever" into "0 of 8, retried automatically."

Topology awareness: gang scheduling isn't enough on its own

Getting all eight pods scheduled somewhere solves the all-or-nothing problem. It doesn't solve the next one: distributed training over NVLink or 400G Ethernet is latency-sensitive enough that where those eight pods land relative to each other changes throughput by a lot, not a little.

Kueue's topology-aware placement understands a zone-over-rack-over-host hierarchy and bin-packs a job's pods onto adjacent nodes rather than scattering them wherever capacity happens to be free. Gang scheduling gets you eight pods running together. Topology awareness gets you eight pods running together fast.

This isn't only an ML problem

The specific example is GPU training because that's where the pain is loudest right now, but the underlying failure mode - a job whose pods have to arrive together to do useful work at all - applies to any tightly-coupled distributed system: MPI jobs, Spark executor pools, anything with a synchronization barrier before real work starts. If your workload has a moment early on where every worker waits for every other worker, the default scheduler's per-pod admission model is working against you, not for you.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

Issue #034 - Gonzo, CloudEvents, and the container that held while the policy didn't

Ilia Gusev — Tue, 08 Sep 2026 14:01:42 GMT

Kubernetes Dashboard is archived. Lens is a subscription. Between those two, one project ended up as the answer to both - which made this week's Showdown block write itself. That's the shape underneath all four of these: something that looked settled - a boundary, a standard, a free tool, a security model - turning out to hold less than advertised.

Grab bag, round two.

🚀 Sandbox Watch: Gonzo

What it is

A terminal log viewer built specifically for OTLP, from ControlTheory, a small observability startup. It surfaced publicly around KubeCon 2025 USA, MIT licensed, sitting around 2,800 GitHub stars now.

How it works

Built on Bubble Tea, Charm's TUI framework - not the tview/tcell stack k9s runs on. "Inspired by k9s" means the 2x2 grid layout and vim-style keybindings, not shared code. It runs its own OTLP receiver - gRPC on 4317, HTTP on 4318 - so logs land directly, no Loki or Elasticsearch sitting in between. It also auto-detects JSON, logfmt, and plain text, with prebuilt integrations for Kubernetes pod logs, Docker, syslog, and a long list of platforms: Vercel, Supabase, Railway, Cloudflare Workers, Netlify, Fly.io, Render, AWS CloudWatch.

The honest part

A Show HN commenter reported 5,000 log lines taking over six minutes to process on their machine - "so slow" was the exact complaint. Worth checking against your own log volume before you commit to it. There's also a fixed bug about returning terminal control to the parent process "(e.g. k9s)" on quit, which tells you people are already running this as a k9s plugin rather than standalone.

And it isn't purely a community project. Gonzo shares a blog and a company with ControlTheory's paid product, an AI-focused observability platform pitched right alongside it. That doesn't make the free tool worse - it's still MIT, still yours to run - but it's worth knowing you're also looking at a funnel.

💎 The Hidden Gem: CloudEvents

What it is

The CNCF's standard envelope for events - not a message bus, not a delivery protocol, just an agreed shape for "here's what happened." Spec v1.0.2, Graduated in the CNCF in January 2024, health score rated "Excellent." Google, Microsoft, IBM, Red Hat, VMware, SAP, and Serverless Inc. all had a hand in it - a vendor consortium, not one inventor.

The envelope

Four required fields: id, source, specversion, type. A handful of optional ones: datacontenttype, dataschema, subject, time. That's genuinely it - small enough to hold in your head, which is the whole point.

Who actually uses it

Knative Eventing is the deepest integration - Broker and Trigger are built directly around CloudEvents. Dapr wraps pub/sub messages in a CloudEvents envelope by default. AWS EventBridge and Azure Event Grid both support it too, but as an opt-in mode through API Destinations or a schema setting - not what you get by default.

Where it stops helping

The spec doesn't touch routing or delivery guarantees - that's left entirely to whatever's actually moving the event around. And dataschema is optional, which means the envelope is standardized but the payload inside it usually isn't. A consumer still can't validate a CloudEvent's actual data without a separate, out-of-band agreement on what that data looks like. CloudEvents solves "what shape is this event," not "what does it mean" or "will it arrive."

⚔️ The Showdown: Lens went paid, Kubernetes Dashboard went dark, one project answers both

The setup

Mirantis has been tightening Lens's licensing since 2023 - marketplace extensions gated, multi-cluster limited, an Enterprise tier added in 2024. Since October 1, 2024, even the free Personal plan lost team features - shared clusters and Spaces now need a Pro or Enterprise subscription.

The one everyone still recommends, and shouldn't

OpenLens - the open-source fork people reach for reflexively - is dead. Last release was v6.5.2 in June 2023. No cease-and-desist, no drama: once Mirantis stopped shipping an open core to fork from, contributions simply stopped. If a "just use OpenLens" comment is more than a year old, don't trust it.

What's actually alive

FreeLens is the real successor - a fresh community fork started in 2024/2025, sitting around 5,300 stars, cross-platform, no proprietary licensing anywhere in it. If you want the original Lens desktop experience without a subscription, this is it now, not OpenLens.

The other half of the story

Kubernetes Dashboard - the official web UI - was archived on January 21, 2026. The repo is read-only. And the project's own farewell note points people toward Headlamp.

Headlamp has been CNCF Sandbox since May 2023, with Microsoft engineers (formerly Kinvolk) actively maintaining it, a monthly release cadence, and both web and desktop builds behind a plugin system. It's the one project that both halves of this story - the paywalled desktop app and the abandoned official dashboard - now point toward.

My read

Terminal-first: k9s, untouched by any of this. Want the original Lens desktop feel for free: FreeLens. Want the option positioned to be the default going forward, with a real team behind it and a plugin ecosystem: Headlamp. Kubernetes Dashboard has dropped out of the running entirely - it's the cautionary footnote now, not a fourth option.

👮 The Policy: the container isn't the boundary that matters

The framework, and where it comes from

"Boundary, Policy, Lifecycle" as three separate questions for AI-agent sandboxing isn't an industry standard - it's from a single blog post by Luis Cardoso, a malware-analysis researcher, published in January. Worth saying plainly, because backlog notes and secondhand summaries have a way of making a good personal framework sound like consensus. It isn't, yet. But it's a genuinely useful way to cut the problem, and this year handed it a lot of supporting evidence.

What it says

Three separate questions, and most sandbox security effort only answers the first. Boundary: can the code physically escape the container, VM, or process? Policy: even fully inside the boundary, what is the agent actually allowed to touch - credentials, network egress, the host filesystem through a mount? Lifecycle: what happens to that access after the task that needed it ends? Most engineering attention goes to boundary. Most real incidents turn out to be policy.

The week that proved it

In July, security researchers documented a string of sandbox escapes across Cursor, Codex CLI, Gemini CLI, Antigravity, and Claude Code - written up as "the week of sandbox escapes." None of them were a broken container boundary. Every one had the same shape: an agent, still fully inside its sandbox, writes a file the boundary was never designed to police - a .vscode task config, a .claude hook, a modified Python virtualenv, git metadata under a nonstandard directory name that slips past path-based rules. Something outside the sandbox later reads and executes that file, unsandboxed. CVE-2026-35603 and CVE-2026-48124 came out of that wave. Anthropic fixed its case quickly; the write-ups describe the other vendors moving slower.

Why the framing matters

The container held in every one of these. Nobody broke out. The failure sat entirely in the second question - policy - which most teams never separately audit, because "we sandboxed it" already feels like the finished sentence.

What to actually check

Not "can the agent escape." Ask what it can still write to, reach over the network, or leave behind after the task ends, even while fully contained. That's three separate audits, not one.

Links

Four different kinds of "that wasn't actually solid." A dashboard that just stopped existing. A standard that only settles the envelope, not the substance. A free tool that's also a sales funnel, which is fine as long as you know it going in. And a sandbox that held perfectly while the policy around it didn't.

The last one is worth sitting with longer than a headline allows. Five different AI coding tools hit the identical failure shape in the same month, and the container was never the part that broke.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The Flux Kustomization that adopted a namespace it never should have touched

Ilia Gusev — Fri, 04 Sep 2026 14:01:17 GMT

Nobody touched the production Redis namespace that week. That was the whole problem with the postmortem: every line of the incident had a completely different name in it, valkey-test, and it still ended with resources gone from redis-system. Four unrelated decisions, each one reasonable on its own, lined up into a chain that let a Kustomization adopt a namespace it had never been told to manage, then delete it again on schedule, exactly as configured.

Nothing here is a Flux bug. Every step is the tool doing precisely what its config says. That's what makes it worth walking through slowly: four defaults compounding, not a single obvious mistake.

The four ingredients

The production Redis Kustomization watched path: ./kubernetes/, not ./kubernetes/production/redis-system/ - the whole tree above it, and not the directory that actually belonged to it. Flux Kustomization doesn't ask which manifests under that path belong to it, either. It takes all of them recursively, by default. Then there's targetNamespace, which doesn't filter what Flux picks up - it force-rewrites metadata.namespace on every namespaced resource it finds, regardless of what namespace the manifest author actually wrote. And prune: true, turned on for exactly the reason you'd turn it on: keep the cluster from accumulating dead resources nobody remembers creating.

Individually these are all sane defaults. path has to point somewhere. Recursion is what makes a directory of manifests useful instead of requiring one Kustomization per file. targetNamespace exists so you can reuse the same manifests across environments. prune is the thing that makes GitOps actually mean "the cluster matches Git," instead of "the cluster is a superset of Git that only grows."

How the adoption actually happened

Someone added a kubernetes/test/ directory for a Valkey experiment, unrelated to the Redis rollout, just organized under the same parent tree. The Redis Kustomization's next reconciliation walked ./kubernetes/, found the new directory, and pulled every manifest in it into its own managed set, including a namespace manifest for valkey-test.

targetNamespace: redis-system then did exactly what it's documented to do. For cluster-scoped resources like the Namespace object itself, the field is a no-op: Flux just created valkey-test as specified, a stray but harmless namespace. For anything namespaced sitting in that same test directory (a Deployment, a Secret), targetNamespace would have silently rewritten its namespace from valkey-test to redis-system, moving a test workload's manifests into the production namespace without anyone editing a single YAML file to say so.

Weeks later, someone cleaned up by renaming kubernetes/test/ to kubernetes-test/, moving it outside the watched path, which reads as the responsible thing to do. On the next reconciliation, Flux compared its desired state (Git, where the adopted resources no longer appeared under the watched path) against actual state (the cluster, where they still existed, because Flux had created them). prune: true did what it's for: deleted everything it had adopted and could no longer see in Git. Namespace/valkey-test, gone. Anything else it had silently claimed, gone with it.

The version where this isn't recoverable

What actually happened was close to the safe end of this failure mode: Flux deleted only what it had wrongly created itself, and the incident resolved as a strange namespace appearing and disappearing.

The version that doesn't resolve cleanly needs one more coincidence: a filename collision. If the wandering test directory had contained a secret.yaml that happened to share a name with a real production secret already managed under redis-system, one of two things happens instead. Either Flux's apply overwrites the real secret's contents with the test manifest's, silently, on the next reconcile, with no alert. Or, worse, the later prune reads the production secret as something it adopted and no longer sees in Git, and deletes the real one.

prune: true doesn't know the difference between a resource it should own and a resource it accidentally started owning. It only knows what's in Git and what's in the cluster under its watch. If the path is wide enough to accidentally include something it shouldn't, prune becomes, in the plainest sense, a weapon pointed at your own cluster by a rule that's doing exactly what it was told.

What actually closes the gap

Never point a Kustomization's path at a parent directory that might one day contain something unrelated. ./kubernetes/ is the mistake, ./kubernetes/production/redis-system/ is the fix: the deepest, most specific directory that contains only that Kustomization's own manifests and nothing else, ever.
If a shared parent path is genuinely unavoidable, a .fluxignore file at that path (same syntax as .gitignore) excludes subtrees like test/, staging/, or *-test/ from being swept up at all.
targetNamespace is a global override with no per-resource awareness, so stop reaching for it by default. A kustomization.yaml with an explicit namespace: field in the directory itself gives you the same result with none of the blast radius, because it only ever touches manifests that are already scoped to that directory.
spec.wait: true makes Flux wait for resources to report ready before considering the reconcile complete, which surfaces adoption-shaped surprises sooner, before they've had a chance to sit quietly for weeks.
A single YAML change won't close this gap on its own. Treat "do any two Kustomizations' paths overlap" as a question worth asking on a recurring basis, the same way you'd audit IAM policies for overlap: the failure mode here is two good configs that were never checked against each other, not a bad config in isolation.

Links

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The 25% of your node that's gone before the first pod lands

Ilia Gusev — Wed, 02 Sep 2026 14:03:06 GMT

Somebody on my team once sized a node pool off the instance label. c5.4xlarge, 16 vCPU, done, that's what goes in the capacity plan. Then we actually looked at what kubectl describe node reported as allocatable, and it wasn't 16. It was closer to 15.5 on paper and noticeably less than that once the DaemonSets landed. Nobody had lied to us. We'd just never asked the node what it thought "capacity" meant.

That gap, between the number on the instance and the number a pod can actually schedule into, is usually 20-30% on a real cluster, and almost none of it shows up on a kubectl top dashboard. It's not a bug. Every layer taking a slice has a legitimate reason to exist. It's just that nobody sums the slices before doing capacity planning, so the plan is wrong by a fifth to a third before the first workload lands.

Here's where it actually goes.

kube-reserved and system-reserved

Every managed Kubernetes offering carves out CPU and memory for the kubelet, the container runtime, and the OS itself before your pods ever see it. The three big ones do it with three different formulas, and none of them show up on the labels on the instance type.

AKS is the most transparent about the math: kube-reserved is the lesser of 20 MB × max-pods + 50 MB or 25% of total memory. An 8 GB node running 30 pods reserves 650 MB before anything else happens. GKE publishes reference numbers too: a 2 vCPU / 7.5 GB node reserves roughly 70m CPU and 1,736 Mi memory as kube-reserved on its own. EKS doesn't publish an exact formula at all; the common community convention baked into most bootstrap scripts is memory=0.3Gi, ephemeral-storage=1Gi for system-reserved, which is a convention people copied into their tooling, not a number AWS commits to.

Three providers, three different answers to "how much of this node is actually mine," and the honest answer is you have to ask the node, not the instance catalog.

Eviction thresholds, and the version cliff nobody warns you about

Reserved capacity is the visible tax. Eviction thresholds are the one nobody budgets for: they carve out headroom that never shows up as "used" on any graph, because it's specifically the memory the kubelet refuses to let you use, held in reserve for the moment things go sideways.

GKE's default memory.available hard eviction threshold is 100Mi. AKS matches that on 1.29 and later. AKS before 1.29 defaulted to 750Mi, seven and a half times more headroom carved out of every single node, silently, as part of a version bump most teams treat as routine. If you fleet-upgraded across that boundary and your capacity math didn't move, it was wrong the whole time before and it's differently wrong now. EKS's commonly configured value in community bootstrap tooling sits around 200Mi, in between the other two, and again: not a published AWS default, a convention.

None of these thresholds are wrong to have. A node with zero eviction headroom OOM-kills workloads instead of gracefully shedding the least important pod first. The reservation itself is fine. What trips people up is that "allocatable" on the node object already accounts for kube-reserved, while the eviction threshold sits on top of that, invisible to anything reading the allocatable field. Most capacity math stops at allocatable and never goes further.

Pause containers: small, until you're at scale

This one is genuinely tiny per pod: a pause container holding the shared network and IPC namespace for everything else in the pod runs around 1 MB of resident memory. Nobody should reorganize a sprint around it.

But it's 1 MB times every pod on every node, all the time, for the lifetime of the cluster, and it's invisible in the same way the eviction threshold is: real memory the OS has committed, attributed to a container nobody thinks about when they're reading a namespace's usage graph. At 1,000 pods that's roughly a gigabyte of RAM doing nothing but holding namespaces open, spread across your fleet in amounts too small to alert on and too persistent to ever go away.

No budget breaks on it alone. What it does is quietly turn "sum of container requests" into a slightly wrong answer to "how much memory does this cluster actually need" - permanently, by a margin that grows with pod count.

DaemonSets: the capacity your FinOps dashboard calls "infrastructure"

This is the one that actually moves budgets, and it's the one most cost dashboards get structurally wrong.

CNI, CSI, the log shipper, the node-level monitoring agent, whatever security or service-mesh sidecar-injector runs as a DaemonSet - every one of them takes a CPU and memory reservation on every node in the cluster, multiplied by node count, for the entire life of the fleet. This cost scales with your node count, not your workloads, which means it moves in lockstep with your other costs, whatever's already driving the bill.

The reservation itself isn't what costs money. Most FinOps tooling attributes CPU and memory by namespace or by workload label, and DaemonSets frequently live in a kube-system or platform namespace that gets bucketed as "infrastructure" and never allocated back to the teams whose CNI policies, log volume, or security scanning drove the agent's resource needs in the first place - that's where the money actually goes missing. Five or six DaemonSets on a hundred-node cluster is a real, attributable cost that shows up on nobody's per-team invoice.

If your FinOps dashboard has a line called "infrastructure" that's bigger than any single team's line, that's usually where it's hiding.

Hosted control plane fees: the one that's actually on your bill

Everything above is invisible capacity tax. This one, at least, is a real line item - it's just inconsistently visible depending on who's cutting your check.

As of this year, EKS and GKE both charge $0.10 per cluster per hour for the control plane while your Kubernetes version is in standard support - about $73/month, though GKE offsets that with a $74.40 monthly credit per billing account that covers exactly one zonal or Autopilot cluster. AKS's free tier charges nothing for the control plane at all; the $0.10/hour tier buys you a 99.95% uptime SLA you don't get for free. And EKS has a second number worth knowing before you need it: once a cluster's Kubernetes version rolls into extended support, that $0.10/hour becomes $0.60/hour - six times the base rate, for the exact same control plane, as the penalty for not upgrading.

On bare metal, none of this exists as a line item, because the control plane is your hardware and your amortization schedule, which is precisely why it's easy to under-budget for it when you migrate the other direction.

What to actually do with this

kubectl top was never lying to you about usage. It was answering a question you weren't asking - "how much is running" instead of "how much of this node's sticker capacity did I actually get to use." Those are different numbers, and the gap between them is kube-reserved plus the eviction threshold plus pause-container overhead plus your DaemonSet count, none of which show up in the same place.

The concrete version: read kubectl describe node and compare Capacity to Allocatable yourself, on your actual node type, on your actual provider - don't trust the instance catalog. Then subtract your DaemonSet requests times node count before you tell finance what a node pool costs per unit of real workload capacity. That number, not the instance label, is the one your capacity plan should be built on.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

Issue #033 - TOON, zot, and the RCE Kubernetes calls working as intended

Ilia Gusev — Tue, 01 Sep 2026 14:03:31 GMT

Five deep-dive weeks in a row, and I promised the next open week would break the pattern. What landed in it doesn't share a vendor or a language, but it shares a shape: each of these four is a boundary someone drew somewhere you wouldn't expect. How much structure an LLM really needs. How much a registry needs to run at all. Where Prometheus Operator's declarative reach quietly stops. And what Kubernetes itself is willing to call a bug.

Grab bag's back.

🚀 Sandbox Watch: TOON

What it is

A serialization format built specifically for LLM context, not for storage or wire transfer. Token-Oriented Object Notation - TOON for short - shipped in October 2025 from a single maintainer, Johann Schopplich, MIT licensed. It hit Hacker News and picked up over 2,000 stars in three days. It's sitting past 25,000 now.

How it works

Not a compression trick. TOON borrows YAML-style indentation for nested objects, then switches to a CSV-like table layout the moment it sees an array of uniform objects - which is most of what actually gets shoved into a prompt: query results, API responses, tool-call output. Each array gets an explicit [N] row count and a {fields} header up front. That's not for humans. It's a guardrail the model uses to check its own parsing as it goes, instead of guessing where one object ends and the next begins from indentation alone.

The numbers, and where they stop working

The project's own benchmark: 46.3% fewer tokens than formatted JSON, at 70.1% accuracy on retrieval tasks. An independent run across four models - Claude Haiku, Gemini Flash, GPT-5 Nano, Grok - found 39.6% fewer tokens, with accuracy that came out higher than JSON's, 73.9% against 69.7%.

The authors are upfront about where it loses, too. Deeply nested or non-uniform data - minified JSON often wins outright. Semi-uniform arrays, somewhere around 40-60% field consistency, and the savings mostly evaporate. Purely tabular data - plain CSV is still smaller, TOON's explicit lengths and field lists cost you 5-10% over that. And on local or quantized models, some setups parse JSON faster in wall-clock time even while burning more tokens to do it.

When to reach for it

Not as a database format, not as an API contract - the authors call it a translation layer for LLM input, and that's the honest framing. If you're stuffing a hundred rows of uniform records into a context window before a completion call, it's worth trying. If your payload is one deeply nested object, don't bother.

💎 The Hidden Gem: zot

What it is

A vendor-neutral, OCI-native container registry. CNCF Sandbox since December 2022, still shipping - v2.1.19 landed in early August. About 2,700 stars, small next to Harbor's, but the commit graph is alive.

Where it differs from Harbor

I went looking for "zot has full OCI Distribution Spec support, Harbor doesn't" as the pitch, and that angle is stale - Harbor implements spec 1.1 too now, so that's not a real differentiator anymore.

The real one: zot runs with no database at all. Harbor needs Postgres behind it. For an edge deployment or anything resource-constrained, that one line is the whole pitch: one static binary, no stateful dependency to babysit. zot also has native support for the OCI Referrers API, which is the mechanism Cosign actually uses to store and look up signatures, so signature and SBOM attachment works as a first-class citizen rather than a bolted-on scan step.

The honest trade

Harbor still has a web UI. RBAC, project-based multi-tenancy, built-in vulnerability scanning, replication between registries, audit logging - and it's CNCF Graduated, zot is Sandbox. None of that is small. zot isn't trying to be a Harbor replacement; it's what you reach for when you want a registry to just serve OCI artifacts and nothing else runs on the same box.

CoreWeave uses zot as the registry behind their CKS Kubernetes service - it's in their own changelog, not a case study somebody paid for.

⚔️ The Showdown: the Probe CRD gap nobody's closed

The comparison I went looking for, and didn't find

I went in expecting a head-to-head: some operator wrapping Prometheus Blackbox Exporter versus using Blackbox Exporter directly. What I found instead is that the project going by that description - endpoint-monitoring-operator, 23 stars - isn't a Blackbox Exporter wrapper at all. It's its own thing: a separate EndpointMonitor CRD, its own probe logic for HTTP, HTTP-with-JSON-body-validation, TCP, DNS, ICMP, even Trino and OpenSearch health checks, and it alerts straight to Slack or email. It doesn't talk to Prometheus. So the comparison as pitched doesn't exist - which turned out to be more interesting than the comparison itself.

Where the real gap is

Prometheus Operator ships a Probe CRD for exactly this job - declaratively wiring blackbox-style checks to Blackbox Exporter, either a static target list or auto-discovered from your Ingress objects. It's mature, it's part of kube-prometheus-stack, most clusters running the standard observability stack already have it.

But it was built for simple, ping-style checks. Go looking at the project's own GitHub discussions and you'll find someone trying to describe hundreds of HTTP targets, each with its own headers and body, and hitting a wall - the Probe CRD has no way to express per-target Blackbox module configuration at that level. A maintainer's own answer in that thread: what's actually missing is "a CRD to deploy and manage Blackbox Exporter" itself, not just targets pointed at one. That issue has been open for years.

My read

That's why people write bespoke operators like endpoint-monitoring-operator instead of extending the Prometheus ecosystem: not out of preference, but because managing Blackbox Exporter's own module config was never made declarative. Anyone whose checks need real assertions - not "did we get a 200," but "does this JSON field say what it should" - ends up walking away from Blackbox Exporter entirely and building something narrower and custom. Nobody built a rival tool here. They just never wrote the CRD that would have made Blackbox Exporter's own config declarative.

👮 The Policy: the RCE Kubernetes calls working as intended

What it takes

This attack starts from one RBAC permission most clusters hand out without a second thought: get on nodes/proxy. No prior foothold on the node, no shell on the cluster required - the permission alone gets you in.

Security researcher Graham Helton published the mechanism in late January. The apiserver's nodes/proxy endpoint is meant to let you read things like node metrics through the apiserver instead of reaching the kubelet directly. The kubelet, on the other end, decides whether to allow a WebSocket upgrade based on the HTTP method of the handshake - and per RFC 6455, that method is always GET. Which means the secondary check that's supposed to gate an actual /exec - the create verb - never fires on the WebSocket path. get on nodes/proxy is enough, by itself, to open a connection to kubelet:10250 and run a command inside any pod on that node.

The 69 charts

Helton didn't stop at the theory - he audited what ships with this permission. Sixty-nine Helm charts grant nodes/proxy get as a matter of course, the full list with links is in his write-up. The names on it are the ones nobody questions: Prometheus, Grafana's Promtail, Datadog, Elastic Agent, Cilium, the OpenTelemetry Collector, Trivy Operator when its cluster role is enabled. Exactly the observability stack most platform teams install without reading the RBAC it asks for.

Working as intended

There's no CVE here, and there isn't going to be one. Kubernetes' Security Response Committee closed it on January 23rd, classified as Won't Fix, working as intended. In their own words, checking a verb at one layer and an HTTP method at another is "brittle, architecturally incorrect" - so instead of patching the specific hole, they're pointing at KEP-2862, Fine-Grained Kubelet API Authorization. It splits the kubelet API into separate, narrower permissions - nodes/metrics, nodes/stats, nodes/log, nodes/healthz, nodes/pods - instead of one nodes/proxy that covers all of it. It's Beta now, behind a feature gate, GA is targeted for April.

Worth being precise about how this differs from the kubelet-API piece I ran back in issue-015: that one assumed an attacker already sitting on the node network, reaching port 10250 directly. This one needs nothing but a ServiceAccount token carrying a permission your monitoring stack probably already has. No node access required at all - a different privilege boundary, not a rerun of the same finding.

What to do about it

Audit which service accounts in your cluster hold nodes/proxy. If you're running any of the standard observability charts, the answer is probably "more than you'd guess." Then treat KEP-2862's rollout as your real timeline, not a patch that's coming - because it isn't.

Links

Four boundaries, four different people drawing them. What TOON decided an LLM doesn't need. What zot decided a registry doesn't need. The CRD nobody at Prometheus Operator got around to writing. And the line Kubernetes drew around what even counts as a bug.

The last one's the one I keep coming back to - not because it got patched, but because it didn't, and it's probably sitting in your cluster right now if you're running anything that watches your nodes.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

LowCardinality: the dictionary stops at 8192

Ilia Gusev — Fri, 28 Aug 2026 14:04:54 GMT

Wrap a string column in LowCardinality and ClickHouse stops storing the string. It builds a dictionary of distinct values and stores small integer positions into it, so a GROUP BY compares integers instead of strings and the column compresses like integers too. On a country code or an HTTP method the win is large and free.

The advice attached to it is always some version of "use it under ten thousand distinct values". That number is real, and it's a rounded-off consequence of a setting most people have never looked at.

low_cardinality_max_dictionary_size = 8192

What the setting does

8192 is the maximum size, in rows, of the dictionary ClickHouse will write for a column. The documentation's blunt about what happens when you exceed it:

All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

No error. No warning. No log line at the default level. Values that fit go in the dictionary as integer keys; everything past the limit is written as plain strings, in the same column, and the column keeps working exactly as before.

There's a companion setting that shapes what "past the limit" means:

low_cardinality_use_single_dictionary_for_part = false

With the default, when a dictionary fills up the server starts a new one rather than giving up. So a part with many distinct values ends up carrying several dictionaries plus whatever spilled over. Set it to 1 and you get exactly one dictionary per part and everything beyond 8192 stored the ordinary way.

Either way the outcome is the same in the direction that matters: the type stops being an optimisation and starts being an overhead, and nothing tells you.

Why the failure is quiet in both directions

The reason this catches people is that the type's behaviour degrades along two axes that don't move together.

Reads degrade gently. A partly-encoded column still answers queries; you just get less of the compression and less of the integer-comparison speedup than you expected, proportional to how much spilled.

Writes degrade immediately. Every insert has to hash the incoming value and look it up in the current dictionary, and that cost is paid whether or not the value ends up encoded. On a column with millions of distinct values you're paying full price for hashing on every row and getting nothing back.

Memory sits in the middle. Dictionaries live in RAM while parts are being read and merged, so a high-cardinality LowCardinality column adds resident memory in exchange for the storage saving it isn't delivering.

The documentation puts the far edge at a hundred thousand distinct values, past which the type "can perform worse in comparison with using ordinary data types". Between 8192 and 100k you're in a zone where it's neither clearly good nor clearly bad, and that zone is where nobody investigates.

Finding out which columns are lying to you

Cardinality per column is one query, and it's worth running against a real table rather than a sample:

SELECT
    uniqExact(status)      AS status_uniq,
    uniqExact(user_agent)  AS ua_uniq,
    uniqExact(request_id)  AS rid_uniq
FROM logs
WHERE event_date >= today() - 7;

Anything above 8192 that's currently declared LowCardinality is spilling. Anything comfortably under it that's a plain String is leaving a win on the table.

The subtlety is that cardinality is measured per part, not per table, and parts are built from inserts and merges. A column with 50,000 distinct values spread evenly across time might sit under 8192 within any single part and encode perfectly well, while the same 50,000 values arriving in one bulk load will not. Sorting order matters here too, since data ordered by the column groups its values into fewer parts.

Which means the honest test is measurement rather than arithmetic:

SELECT
    name,
    formatReadableSize(sum(data_compressed_bytes))   AS compressed,
    formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed,
    round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 1) AS ratio
FROM system.columns
WHERE table = 'logs' AND database = currentDatabase()
GROUP BY name
ORDER BY sum(data_compressed_bytes) DESC;

A LowCardinality column whose compression ratio looks like its neighbouring plain strings isn't being encoded.

Enum is not the alternative people think it is

Enum8 and Enum16 do the same job with the value list fixed in the DDL, which makes them faster on insert - the mapping is static, so there's no hashing and no dictionary to grow - and completely rigid. A value that isn't in the enum is an error, and adding one means an ALTER TABLE.

That rigidity is a feature for a closed set: days of the week, a payment state machine you control, HTTP methods. It's a production incident for anything a third party can extend, because the day a new value appears your inserts start failing.

Enum16 also tops out at 65,535 values, which is comfortably above the point where LowCardinality has stopped helping. If you're choosing between them on capacity you've already picked the wrong tool for that column.

The limits worth knowing

LowCardinality isn't free at query time in every direction. Most string functions work through it transparently, but anything that has to materialise the actual strings - tokenisation, some regex paths, certain joins - decodes the whole column first, and for those the type costs you rather than saving you.

The setting is per-insert rather than per-column, which surprises people who go looking for a way to raise the limit on one wide column. Raising low_cardinality_max_dictionary_size globally trades RAM across every LowCardinality column in the server for the benefit of one, so the usual answer is to change the column type rather than the setting.

And the guidance shifts with version: this is ClickHouse 26.7 behaviour, and defaults in this area have moved before. Read the value out of your own server rather than trusting a number in a post:

SELECT name, value FROM system.settings WHERE name LIKE 'low_cardinality%';

Knative and PDB: protects nothing, blocks drains forever

Ilia Gusev — Wed, 26 Aug 2026 14:01:31 GMT

The advice sounds reasonable and it's everywhere: don't put a PodDisruptionBudget on a Knative service, because minAvailable: 1 will block scale-to-zero and you'll lose the thing you adopted Knative for.

It will not block scale-to-zero. It cannot. And that is worse, because people set it expecting protection and get a pod that disappears anyway plus a node they can no longer drain.

Why the autoscaler doesn't care

A PodDisruptionBudget is enforced in exactly one place: the Eviction API. When something calls POST /pods/{name}/eviction, the API server consults every PDB whose selector matches that pod and refuses if the budget would be violated. That is the entire mechanism.

Scaling down is not an eviction. When the Knative Pod Autoscaler decides a revision has no traffic, it writes to the Deployment's scale subresource - replicas: 0 - and the ReplicaSet controller deletes the pods directly. No eviction call, no PDB lookup, no budget check.

So the sequence people expect - PDB says one pod must stay, therefore Knative can't go to zero - never happens. The pod goes away on schedule and the PDB is not consulted at any point.

You can watch this from two terminals if you want to be sure. Send traffic, watch a pod appear, stop traffic, and watch kubectl get pdb report ALLOWED DISRUPTIONS 0 the whole time the pod is being removed underneath it.

What it does block

Node drains, which is precisely what a PDB is for, and precisely the interaction nobody plans for.

Take a service pinned with autoscaling.knative.dev/minScale: "1" because it can't tolerate cold starts, and a PDB with minAvailable: 1 because that's what you always write. There is one pod. The budget allows zero disruptions. kubectl drain on that node calls the Eviction API, gets a 429, retries, and keeps retrying.

Forever. The autoscaler has no reason to add a second pod, because there's no traffic pressure to justify one, and the drain has no way to ask for one. A cluster upgrade rolling through the fleet stops on that node and waits for a human.

That failure has a shape worth recognising: a PDB that permits zero disruptions on a workload that will never scale up on its own is a permanent drain block, and it is silent until the day somebody drains that specific node.

Then there's the revision problem

Even where a PDB is the right tool, Knative's object model fights the way you'd normally write one.

Revisions are immutable. Every configuration change makes a new Revision, each with its own Deployment and its own pod labels:

serving.knative.dev/revision: my-app-00042
serving.knative.dev/service:  my-app

Write a PDB against the revision label and it's dead weight the moment you deploy: it now selects a Revision that's scaling to zero while the new one runs unprotected. Keeping up means creating a PDB per deploy and garbage-collecting the old ones, which is a controller nobody wants to own.

Selecting on serving.knative.dev/service covers every revision of a service at once, including both sides of a traffic split, which is usually what you meant. It also means your budget arithmetic has to account for pods across two revisions during a rollout.

Where Knative's availability comes from

For a service that scales to zero, asking "how do I keep a pod available" is the wrong question, because the answer is that there isn't one and that's the design.

Requests to a zero-scaled revision don't fail. They go to the Activator, which buffers the request, tells the autoscaler to scale up, and forwards it once a pod is ready. Your availability during that window is the Activator's availability, not your pod's.

Which is why the PDBs that matter for a Knative install are the ones on Knative's own components, and they already exist. A default install runs the activator, autoscaler, controller and webhook with two replicas, and ships an activator-pdb with minAvailable: 70%. If you're going to spend attention on disruption budgets in a Knative cluster, spend it verifying those survived your install tooling rather than adding new ones to application revisions.

What to do instead

For most services: nothing. Scale-to-zero and the Activator are the availability story, and a PDB adds a drain hazard in exchange for no protection.

For a service that must always be warm, set minScale: 2 rather than 1. Two pods across two nodes give a drain somewhere to move work to, and only then does a budget make sense - written as maxUnavailable: 1 rather than minAvailable: 1, because that shape always permits one disruption regardless of how the replica count moves under you.

If you inherited a cluster and want to know whether this is already waiting for you:

kubectl get pdb -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed \
  | awk '$3=="0"'

Anything listing zero allowed disruptions is a node drain that will hang. Cross-check those against Knative services with minScale: 1.

The limits worth knowing

PDBs only ever governed voluntary disruptions. A node that dies takes your pod with it whatever the budget says, and no arrangement of PDBs changes that - the guarantee was always about planned maintenance.

Knative also isn't unique here. Any autoscaler that writes the replica count directly, HPA included, moves pods without consulting a PDB. The Knative case stands out because scale-to-zero makes the gap between "budget says one" and "reality says none" a routine daily event rather than an edge case.

And this is Knative Serving 1.23 behaviour. Component defaults, including the activator PDB, are the kind of thing that shifts between releases, so check yours rather than trusting a number in a post.

Issue #032 - Keyless signing: nobody manages signing keys anymore

Ilia Gusev — Tue, 25 Aug 2026 14:02:01 GMT

For about twenty years, code signing meant key custody. Generate a key, protect it for its lifetime, and accept that losing it costs you your identity while leaking it costs everyone else theirs. Every practice around it - HSMs, air-gapped signing machines, elaborate ceremonies - existed because the key was a long-lived secret that had to stay secret.

That problem is gone. Not mitigated, gone: in a Sigstore signature there's no key to keep, because the key is generated, used once, and thrown away inside the same command.

Which is a genuine win, and it moved the failure somewhere less visible. I covered Sigstore as one block in issue #005 back in February, at the level of "cosign signs your images". This is the other half: what you're actually trusting once the key is gone, and why the interesting bugs now live in the verify path rather than the sign path.

🏗️ Architectural Pattern: a certificate that expires before you finish reading this

The exchange

The flow is short enough to hold in your head.

You authenticate to an OIDC provider - Google, GitHub, your own issuer - and get an ID token. Cosign generates a throwaway keypair locally. It sends the public key and the token to Fulcio, the CA. Fulcio validates the token against the provider, then issues an X.509 certificate binding that public key to the identity in the token: the email or workflow reference goes in the Subject Alternative Name, and the OIDC issuer goes in a dedicated extension.

The certificate's valid for ten minutes.

You sign, the private key is discarded, and the signature plus the certificate go into a Rekor entry, which timestamps the fact that this certificate signed this digest while the certificate was still valid.

Why ten minutes is the point

A ten-minute certificate can't be revoked in any meaningful sense, and it doesn't need to be. Revocation infrastructure - CRLs, OCSP, the whole apparatus - exists because certificates outlive the circumstances that justified them. Shrink the window below the time it takes to notice and respond to a compromise and the problem stops being worth solving.

The cost is that a ten-minute certificate is useless as a verification anchor on its own. By the time anyone checks your signature, the certificate expired months ago. So verification can't ask "is this certificate valid now". It asks "was this certificate valid when the signature was made", and the only reason that question has an answer is the transparency log entry that timestamped it.

That inverts a dependency people don't expect. In classic signing, the log is an audit nicety. Here it's load-bearing: without a log entry, an expired certificate proves nothing.

What you're trusting instead

The key is gone and the trust didn't evaporate, it moved. A verifier is now trusting, in order: that the OIDC provider correctly authenticated whoever asked; that Fulcio correctly validated the token; that the log entry is genuine and inclusion-proven; and that the identity in the SAN is one you meant to accept.

The last one is yours. The first three belong to somebody else, which is fine - that's what a public good looks like - but it's worth saying plainly, because "keyless" gets read as "trustless" often enough to matter.

🆚 The Showdown: Rekor v2 against the log it replaced

Rekor v2 reached GA this year, and it's an unusual upgrade in that most of the release notes are things being taken away.

Gone: signed timestamps returned by the log. Gone: the search index. Gone: entry types intoto, rekord, helm, tuf, rfc3161, jar, rpm, cose and alpine, leaving hashedrekord for artifacts and dsse for attestations. Gone: attestation storage, so you keep your attestations next to your artifacts like any other file.

Underneath, the Trillian backend was replaced with a tile-based log built on Tessera, and each shard now has its own URL along the lines of logYEAR-rev.rekor.sigstore.dev rather than hiding behind one endpoint.

The case for deleting things. A transparency log's only job is to make it impossible to sign something without leaving a permanent, publicly auditable trace. Every additional feature - a search index, ten entry formats, blob storage - is surface that has to stay correct forever, because a transparency log you have to migrate off is not much of a transparency log. Cutting to two entry types and a tile-based backend makes it cheap enough to run that other people can run their own, which is the actual goal.

The case against, and it's real. The search index going away means you can no longer ask the log "show me everything signed by this identity" without building that index yourself. For anyone doing detection - watching for signatures made by an identity that shouldn't be signing - that was the useful interface, and its replacement is "coming later as a separate verifiable service".

The client-side consequence is the one to plan for: shard URLs must come from the TUF root, not from your config. Hardcode a log URL today and you've built a thing that breaks on the next rotation. Rekor v1 keeps running in parallel and will eventually freeze for new uploads, with a year's notice.

👮 The Policy: the verify command is the whole security boundary

Signing is now easy enough that people do it. Verification is where the work moved, and it's the half that gets copy-pasted.

Cosign v2 made two flags mandatory for keyless verification, and the reason is worth internalising:

cosign verify \
  --certificate-identity 'https://github.com/acme/api/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/acme/api:1.4.2

Without those, "verify" means "some identity that Fulcio was willing to certify signed this", which is every GitHub account in existence. The command would succeed on an image signed by an attacker with a free account, and it'd look exactly like success.

The regex habit is the same bug wearing a hat. --certificate-identity-regexp='.*' shows up constantly, usually added to make a pipeline go green. It restores precisely the property the mandatory flags were introduced to remove. If you need a pattern, anchor it: pin the issuer exactly, and let the regex cover only what varies, like a tag or a branch under a repository you control.

Pin the workflow, not the repository. For CI identities the SAN contains the workflow file and the ref. Accepting any workflow in a repository accepts the one an attacker adds in a pull request, so the identity you pin should name the release workflow on the branch you protect.

The verification path is where the bugs are

If that sounds theoretical, January settled it. CVE-2026-22703 let a crafted bundle verify successfully even though its embedded Rekor entry didn't reference the artifact's digest, signature or public key. Cosign checked the log entry's signature and skipped the comparison that ties the entry to the thing in front of it, so any valid entry would do.

The practical effect is that an attacker who already had your identity could sign something and attach a Rekor entry pointing at an unrelated event, which removes the audit trail - the one property the log exists to provide.

Two details make it more instructive than an ordinary CVE. It affected cosign v2 users on default flags, while v3 users on default flags were unaffected. And it was a regression: the same class of bug had been fixed before and came back through a refactor.

Fixed in v2.6.2 and v3.0.4. Current releases are v3.1.3 and v2.6.5, and if your CI's pinned to a cosign version from last year, that's this week's fifteen-minute job.

The honest part

None of this makes an artifact good. A verified signature says a particular identity signed a particular digest at a particular time, and that's all it says. A compromised CI account produces perfectly valid signatures over malicious images.

What it buys you is attribution and an audit trail nobody can edit without being seen, which is a large improvement over an unsigned artifact and a smaller one than the word "verified" suggests.

And the monitoring gap is real. Sigstore's value proposition assumes somebody notices when an identity signs something it shouldn't. With the search index gone from Rekor v2, the tooling for that is thinner than it was, so if your threat model depends on detection rather than on after-the-fact forensics, that's a project rather than a flag.

Where the difficulty went

The thing I keep noticing about this transition is that it didn't remove difficulty, it relocated it into a place with worse feedback.

Key custody was hard, and it was hard in a way that announced itself. Everyone knew the HSM was a chore. Nobody accidentally believed they had good key management when they didn't.

Identity-based verification is easy to get wrong and comfortable to leave wrong, because a misconfigured verify command returns zero and prints a checkmark. The pipeline is green, the dashboard says signatures are enforced, and the policy admits anyone with a GitHub account. There's no chore to remind you.

We checked ours after reading the January advisory. Two of nine pipelines pinned issuer and identity properly. Five used a regexp with .* somewhere in it. Two verified with --insecure-ignore-tlog, which somebody added during an incident in March and nobody removed.

Questions? Feedback? Reply to this email. I actually read them.

Ilia

Topology-aware routing: 7 replicas work, 8 don't

Ilia Gusev — Fri, 21 Aug 2026 14:01:48 GMT

Cross-zone traffic is billed per gigabyte, so the annotation that keeps requests inside one availability zone looks like free money:

metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto

You apply it, the bill doesn't move, and there's nothing in kubectl describe service that looks like an error. The usual conclusion is that the feature is broken or that the cluster is missing zone labels.

Neither. The controller looked at your service, decided that routing by zone would overload something, and declined. It even said so, in an event nobody reads.

What the controller is actually deciding

The EndpointSlice controller doesn't route anything. It writes hints into EndpointSlices - a note on each endpoint saying "this one is for zone X" - and kube-proxy on each node honours the hints by restricting its endpoint set to the local zone.

It decides who gets a hint from allocatable CPU per zone, and pod counts never enter into it:

The controller allocates a proportional amount of endpoints to each zone. This proportion is based on the allocatable CPU cores for nodes running in that zone.

A zone with twice the CPU is expected to receive twice the traffic, so it should hold twice the endpoints. From that expectation the controller computes, per zone, the smallest endpoint count it would tolerate:

const overloadThreshold float64 = 0.2

desired := ratio * float64(numEndpoints)
minimum := int(math.Ceil(desired * (1 / (1 + overloadThreshold))))

Read that as: each endpoint may end up carrying at most 20% more than its fair share. Anything worse and the controller refuses. It sums minimum across zones, and if the total exceeds the number of endpoints you actually have, it gives up and leaves the service on cluster-wide routing.

overloadThreshold is a package-level constant and it isn't exported. There's no flag, no annotation value, no per-service override.

Adding a replica can turn it off

Because minimum is rounded up per zone, the outcome isn't monotonic in replica count. Adding a replica can turn the feature off.

Three zones with equal allocatable CPU, varying the number of ready endpoints:

 3  hints
 4  refused
 5  refused
 6  hints
 7  hints
 8  refused
 9  hints
10  hints
11  refused
12  hints

Eight is the one that catches teams, because eight is a normal number of replicas and seven works. Walk through it: each zone's fair share is 8/3 = 2.67 endpoints, the minimum is ceil(2.67 / 1.2) = 3, three zones need 9, and you have 8. Refused.

Nine replicas works, which is where the "at least 3 per zone" rule of thumb comes from. That rule is a decent summary of the safe region and a bad description of the boundary, and the boundary is where the surprise lives - an HPA that scales 7 to 8 silently disables zone routing, then re-enables it at 9.

Unequal CPU across zones shifts the whole pattern. On a cluster where one zone runs bigger nodes, the working counts differ from the table above, so treat it as a shape rather than a lookup.

Finding out which case you're in

The controller emits a Warning event on the Service with reason TopologyAwareHintsDisabled, and the message names the specific reason:

InsufficientNumberOfEndpoints - fewer endpoints than zones
MinAllocationExceedsOverloadThreshold - the arithmetic above didn't work out
NodesReadyInOneZoneOnly - only one zone has ready nodes
NoZoneSpecified - an endpoint's node has no zone label

So the first diagnostic step is not tcpdump:

kubectl get events --field-selector reason=TopologyAwareHintsDisabled -A

If nothing comes back and the hints still aren't there, check whether the hints exist but kube-proxy is ignoring them:

kubectl get endpointslice -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.zone}{" -> "}{.hints}{"\n"}{end}'

Empty hints with no event usually means the controller never considered the service at all - the annotation is misspelled, or it's on the Deployment instead of the Service.

The newer field is a different mechanism

spec.trafficDistribution shows up in every discussion of this, and reading it as a tidier spelling of the annotation is a mistake worth avoiding.

They're separate code paths in the same controller. The annotation goes through the capacity heuristic described above. trafficDistribution goes through its own reconciler, whose entire policy is one comment in the source:

update adds a same zone topology hint for all ready endpoints

Every ready endpoint gets a hint for its own zone. No CPU ratios, no overload threshold, no minimum endpoint count, no refusal path. PreferSameNode does the same thing at node granularity.

Which means the migration people describe as cosmetic changes the behaviour in both directions. You gain a feature that actually turns on at 8 replicas. You lose the guard that was protecting a thin zone from taking a full share of traffic with one pod in it.

There's a precedence rule too, and it's silent: the reconciler computes canUseTrafficDistribution as valid field and annotation absent. Leave the old annotation on a Service while adding the new field, and the field is ignored entirely. No event, no warning, no status. The source carries a note that the whole branch goes away once the annotation is deprecated under KEP-4444, so the annotation is on its way out - but while both exist, the old one wins.

The assumption underneath all of it

Both mechanisms allocate endpoints by where capacity is. Neither knows where traffic comes from.

That's fine when request origin roughly tracks capacity, which is the common case for internal service-to-service calls in a balanced cluster. It falls apart when a single zone originates most of the traffic - a batch job, a cron-driven fan-out, an ingress tier that isn't spread the way your workloads are. Then zone-local routing concentrates that load onto one zone's endpoints while the others idle, and the cost saving comes out of your tail latency.

This is also the second half of a two-hop story. externalTrafficPolicy governs the first hop, from the external load balancer to a node, and it has its own skew problem. Topology hints govern the second hop, from the node to a pod. They compose, and they fail independently.

The limits worth knowing

Hints are ignored entirely when internalTrafficPolicy: Local is set on the Service. The two features want the same decision made in different places, and the node-local policy wins.

The controller excludes control-plane nodes from its CPU ratios, ignores unready nodes, and pays no attention to taints or tolerations. A zone whose nodes are tainted so your workload can't schedule there still counts toward that zone's expected share, which is a good way to get refusals you can't explain from the pod distribution alone.

And if any single endpoint is missing a hint while others have them, kube-proxy falls back to all endpoints for that service. Partial hints aren't partially applied - it's all or nothing, per service, on every node.

Informer resync: the API call that never happens

Ilia Gusev — Wed, 19 Aug 2026 14:01:18 GMT

The controller was acting on stale data, so somebody lowered resyncPeriod from ten minutes to thirty seconds. It didn't help. Then to five seconds, which didn't help either, and made the process burn CPU on a schedule.

Resync doesn't fetch anything. From the client-go docs on AddEventHandlerWithResyncPeriod:

The resync operation consists of delivering to the handler an update notification for every object in the informer's local cache; it does not add any interactions with the authoritative storage.

Every word of that is load-bearing, and the last clause is the one that gets skipped.

What an informer is made of

Four pieces, and they're easier to reason about named:

A reflector does one LIST against the API server, then opens a WATCH from the resourceVersion that LIST returned. It's the only component that talks to the API server at all.

A DeltaFIFO queues what the reflector produces, as typed deltas: Added, Updated, Deleted, Sync.

An Indexer is the local cache - a thread-safe store with secondary indexes, so your controller can ask for "all pods on node X" without a query leaving the process.

Event handlers are your code, called as deltas drain out of the queue.

Everything a controller reads comes from the Indexer. client.Get() through a cached client, a lister, the informer's GetStore() - all of it is memory, populated by that one watch stream. That's the whole point of the pattern: one watch per resource type per process, shared across every controller in the binary, instead of N controllers polling.

What a resync actually does

The current client-go handles a full resync in five lines, and they answer the question completely:

case SyncAll:
    objs := clientState.List()
    for _, obj := range objs {
        handler.OnUpdate(obj, obj)
    }

clientState is the local store. OnUpdate(obj, obj) passes the same object as both the old and the new value.

So a resync walks your cache and calls your update handler once per object, with old and new identical. Nothing is fetched. Nothing is compared. If your handler starts with a check like "did the spec change since the last version" it will find that it didn't, every time, for every object.

Two consequences fall out of this immediately.

A handler that diffs old against new does nothing on resync. If you wrote your controller to skip work when nothing changed - which is a good habit - resync is a no-op for you, and turning it down will never make anything fresher.

A handler that enqueues unconditionally does full work on resync. Cache of fifty thousand pods, resync every thirty seconds, and you've signed up for fifty thousand reconcile enqueues every thirty seconds. The API server sees none of it. Your CPU graph sees all of it.

Then what is it for

Resync exists for drift between your cache and something the cache can't observe.

A controller that creates cloud load balancers can't see someone deleting one in the console. No Kubernetes object changed, so no watch event fires, and the controller has no reason to look. A periodic resync re-runs reconciliation for everything and gives it that reason.

That's the honest use case, and it's why the sensible values are minutes, not seconds. Ten to thirty minutes covers the drift case. Anything shorter is usually somebody trying to fix a freshness problem that resync has never addressed.

If you have no external state to drift against, zero is a legitimate setting. Plenty of controllers run with resync disabled and are correct, because the watch already delivers every change to every object they care about.

When the data really does go stale

Two failure modes, and neither is fixed by resync.

The watch broke and the reflector is re-listing. Watches die - connection resets, apiserver rollouts, a resourceVersion too old for the watch cache to serve. The reflector handles this by re-listing and starting a new watch, and during that window your cache is whatever it was. Watch HasSynced, which the source is explicit about:

HasSynced returns true if the shared informer's store has been informed by at least one full LIST of the authoritative state of the informer's object collection. This is unrelated to "resync".

You read your own write. You update an object through the API and immediately read it back from the cache. The write went to etcd; the cache updates when the watch event arrives, which is milliseconds later but not zero. Controllers that assume read-after-write against a cached client have a race, and it's the most common informer bug I've seen. If you need the object you just wrote, either read through an uncached client or structure the reconcile to be re-entrant and let the next event carry the new state.

The expensive part is the LIST

Resync is free for the API server. The initial LIST is not.

Historically the apiserver assembled the entire collection in memory before sending a byte of it, so a controller restarting against a hundred thousand pods produced a large allocation spike on the control plane, and a cluster full of controllers restarting together produced several at once.

That's what streaming lists fix. Under KEP-3157 the initial state arrives as a watch that streams items individually, giving the apiserver constant memory overhead instead of one proportional to the collection. Server-side it's the WatchList gate; client-side it's WatchListClient, which reached beta and is on by default. Both are beta rather than GA, so it's worth knowing which of your controllers actually use it before you take credit for the memory graph.

The apiserver side of this story - how watch requests get served without touching etcd at all - is the watch cache, and it's a separate mechanism from anything in this piece.

The limits worth knowing

Resync periods are per handler, but not freely. A shared informer has one resync check period, and a handler asking for a shorter interval than that gets the informer's, not its own. Requesting two seconds on an informer configured for ten minutes gets you ten minutes.

Shared informers are also shared in a way that bites: the informer factory dedupes by resource type, so two controllers asking for pod informers get the same one, along with each other's resync behaviour and each other's cache memory. That's usually what you want and it's occasionally why a memory number doesn't match anyone's mental model of who allocated it.

And the cache holds full objects. An informer on pods in a large cluster is holding every pod spec and status in your process. Transform functions on the informer let you strip fields you'll never read before they hit the store, which is the cheapest memory win available to most controllers and one almost nobody applies.

Issue #031 - Ambient mesh: the sidecar was a five-year detour

Ilia Gusev — Tue, 18 Aug 2026 14:02:54 GMT

Ambient mode went GA in Istio 1.24, in November 2024. That's twenty-one months ago. Istio is on 1.30.3 now, and somewhere in that stretch the question quietly flipped: it stopped being "should we try ambient" and became "why are we still injecting sidecars".

I covered ambient in issue #002 back in January, at the level of the sidecar tax - one Envoy per pod, fifty to a hundred megabytes each, startup ordering problems. That framing is fine and it's also the least interesting thing about the architecture. The resource number is a consequence. The mechanism underneath is stranger than "we moved the proxy to the node".

🏗️ Architectural Pattern: how a node proxy pretends to be a sidecar

The file descriptor trick

The obvious way to build a node-level proxy is to run it on the node, in the host network namespace, and route pod traffic to it. That design has a well-known problem: once the packet leaves the pod, you've lost the pod. Source addresses get rewritten, the proxy has to reconstruct which workload sent what, and every policy decision downstream depends on that reconstruction being right.

Ambient does something else. When a pod joins an ambient-enabled namespace, the istio-cni node agent enters the pod's network namespace, installs redirection rules there, and then hands ztunnel a file descriptor referring to that namespace over a Unix socket.

Ztunnel uses the descriptor to open listening sockets inside the pod's network namespace. One logical proxy instance per pod, sockets that live in the pod's own netns, a process that lives on the node.

So from the pod's point of view the proxy is local. From the node's point of view there's one Rust process instead of a hundred Envoys. The pod identity never has to be reconstructed, because the traffic never left the pod's namespace unattributed.

Three ports do the work, and they're worth knowing when you read a ss output at 2am:

15001   outbound, before HBONE encapsulation
15006   inbound plaintext
15008   inbound HBONE

Inbound TCP is redirected to 15008 or 15006 depending on whether it arrived encrypted. Outbound goes to 15001 and leaves the node wrapped in HBONE, which is HTTP CONNECT over mTLS - a tunnel that carries the original destination as connection metadata rather than as a rewritten address.

The identity problem, and the fix

In the sidecar model, the Envoy holding a workload's certificate ran in that workload's pod. The blast radius of a compromised proxy was one pod, and the certificate it held was its own.

Ztunnel holds a certificate for every unique service account among the pods on its node. It has to - it terminates mTLS on their behalf. A node running forty pods across twelve service accounts has one process holding twelve workload identities.

The guardrail is on the CA side. Ztunnel authenticates to the control plane as itself and then requests certificates for other identities, and the CA enforces that it may only do that for identities actually scheduled on that node. Requests for identities not running there are rejected.

That's a sound design and it is a genuinely different threat model from a sidecar. It's node-scoped rather than pod-scoped, and the thing standing between a compromised ztunnel and a cluster-wide identity grab is a control-plane check rather than the absence of the key material. If your security review asked "what changed", that's the answer, and it's better stated plainly than discovered later.

Where waypoints sit

Ztunnel does L4 and only L4. mTLS, L4 authorization, TCP-level telemetry. It doesn't parse HTTP headers, by design - that's how it stays small and how it stays Rust.

Anything at L7 needs a waypoint, which is a real Envoy, deployed as a normal Deployment for a namespace or a service account. The path becomes:

source pod -> source ztunnel -> waypoint -> destination ztunnel -> destination pod

Four hops where a sidecar mesh had two. That's the trade you're making for the resource savings, and whether it matters depends entirely on how much of your traffic actually needs L7 treatment.

🆚 The Showdown: what the sidecar was still doing for you

The migration guides are cheerful. The list below is the one I'd want before committing, because every item on it is a thing that worked yesterday and needs a decision today.

Per-workload EnvoyFilter and WASM. In sidecar mode you could attach a filter to one workload, because that workload had its own Envoy. Ambient has no per-pod Envoy to attach to. Filters either become global or move into a waypoint, and a waypoint serves a namespace or a service account rather than a single pod. Teams with one bespoke filter for one service feel this immediately.

L7 telemetry by default. A sidecar emitted HTTP metrics, logs and traces for everything passing through it, because everything passing through it was already parsed. Ztunnel gives you L4: bytes, connections, TCP-level authorization decisions. Your request-rate-by-response-code dashboard goes quiet for any workload without a waypoint. This is the single most common surprise, and it surprises people after the migration rather than during it.

Locality-aware routing. A sidecar carried locality information and preferred endpoints in its own zone without being asked. In ambient, traffic that goes through a waypoint goes to wherever that waypoint is, so a waypoint in the wrong zone turns a same-zone call into a cross-zone one - with the latency and the inter-AZ transfer bill that implies. The fix is deploying a waypoint per zone, which is fine, and which nobody plans for on the first pass.

VirtualService. Gateway API is the preferred routing surface in ambient. VirtualService support is alpha there. If your routing rules are years of accumulated VirtualService, that's a translation project, not a label change.

Multi-cluster. Stable in sidecar mode. Available in ambient with limitations that move release to release, which is the polite way of saying check the notes for your exact version rather than trusting a blog post.

And the resource claim deserves an honest reading. "Up to ninety percent" is real arithmetic under the right conditions: it's a ratio between one node proxy and N sidecars, so it scales with pod density and it shrinks the moment you deploy waypoints, because a waypoint is an Envoy again. A cluster with thirty pods a node and L7 policy everywhere saves much less than a cluster with a hundred pods a node running L4 mTLS only. Both are ambient. The savings differ by an order of magnitude.

The failure mode nobody mentions

A sidecar failing takes down one pod. Ztunnel is a DaemonSet, and a ztunnel restart interrupts every meshed pod on that node.

In practice this is mostly fine, because ztunnel restarts are fast and connections re-establish. It's still a different operational shape, and it belongs in the same conversation as the identity scoping: you traded per-pod failure domains for per-node ones, in exchange for not having to restart every application pod to upgrade the mesh.

👮 The Policy: migrating without a flag day

The good news is that the migration is genuinely incremental, and not in the marketing sense. Sidecar pods and ambient pods coexist in one mesh. That means the unit of migration is a namespace, and the rollback is removing a label.

A sequence that works:

Start by turning on L4 for one namespace and changing nothing else. Label it, and you get mTLS and L4 authorization with no pod restarts and no waypoint. This step costs almost nothing in resources and it's where most of the security benefit lives. If you stop here permanently for most namespaces, that's a legitimate end state rather than a half-migration.

Then work out which services need L7. Not which ones have an Envoy today - which ones use a feature that requires parsing HTTP. Header-based routing, JWT claims in authorization policy, retries and timeouts on HTTP semantics, request-level telemetry that someone alerts on. Usually a much shorter list than the number of workloads currently running sidecars.

Deploy waypoints for those and size them like anything else you own. A waypoint is a Deployment: it needs requests, limits, an HPA if the traffic warrants it, and a copy per zone if you care about locality. Treating it as invisible infrastructure is how you get a mesh-wide latency regression out of one under-provisioned pod.

Sort out the dashboards before the cutover, not after. Know which panels are fed by L7 metrics from sidecars, and decide for each one whether it moves to a waypoint or gets rebuilt on L4 signals. Leaving it until afterwards buys you a window where nobody can see anything, and that window is usually when somebody asks whether the migration broke something.

The honest part

This isn't free and it isn't instant. The resource win is real, the operational win of not restarting pods to upgrade the mesh is real, and both are paid for with an extra hop for L7 traffic, a node-scoped identity surface, and a translation project for anyone deep in VirtualService or per-workload filters.

There's also a version of this decision that ends with "neither". If you're running a mesh for mTLS and nothing else, and you're on a cloud CNI that can do transparent encryption between nodes, the honest comparison isn't sidecar against ambient - it's mesh against no mesh. Ambient makes the mesh cheap enough that the question is worth asking again, which is not the same as answering it yes.

What the detour was actually for

Calling the sidecar a five-year detour is unfair in one direction and accurate in the other.

It was unfair because the sidecar proved the model. Transparent mTLS, policy that doesn't live in application code, telemetry nobody had to instrument - all of that had to be demonstrated before anyone would accept a new component in the datapath at all, and the sidecar demonstrated it in the only way available at the time. The kernel primitives ambient depends on weren't there in 2018.

It's accurate because the cost model was wrong from the start, and everyone knew it. One proxy per pod is O(pods) for a job that's O(nodes) plus a bit of L7. We spent five years paying a linear tax for a constant-ish problem, and building tooling to make the tax bearable: injection webhooks, startup ordering, the whole holdApplicationUntilProxyStarts genre of workaround.

The pattern generalises past meshes, and it's the thing I'd take away if I only took one. When a piece of infrastructure needs a lot of tooling to make its cost model tolerable, the tooling is a signal about the cost model, not a solution to it. Sidecar injection webhooks were a very good answer to a question that shouldn't have been asked.

I went and counted our own sidecars after Istio 1.30 shipped. Two hundred and eleven Envoys, of which eleven are attached to workloads that use a single L7 feature. The other two hundred are doing mTLS.

Questions? Feedback? Reply to this email. I actually read them.

Ilia

externalTrafficPolicy Local: client IP for skewed load

Ilia Gusev — Fri, 14 Aug 2026 14:00:30 GMT

Every access log said 10.244.1.x. Rate limiting was useless, the geo rules did nothing, and the abuse dashboard showed one very busy internal address.

The fix is one line, and everyone finds it within an afternoon:

spec:
  externalTrafficPolicy: Local

Real client addresses come back immediately. What arrives later is a load distribution nobody asked for.

Why the addresses were wrong

Default policy is Cluster. Traffic hits any node, and kube-proxy forwards it to a pod wherever that pod lives.

That forward is the problem. If the packet arrives at node A and the pod is on node B, node B's reply has to travel back through node A, because node A is the one holding the connection with the load balancer. To make the return path work, node A rewrites the source address to its own. Your pod sees the node, not the client.

Local tells kube-proxy to stop forwarding. If there's a pod for this Service on the node the packet landed on, deliver it locally, unmodified. If there isn't, drop it. No cross-node hop means no return-path problem means no rewrite, and the client address survives.

The dropped-packet half sounds alarming and mostly isn't, because cloud load balancers health-check the NodePort. Nodes without a local pod fail the check and stop receiving traffic. The mechanism is doing exactly what it says.

The skew

Here's what the health check can't fix.

The load balancer distributes across the nodes that pass the check, and it distributes evenly, because it has no idea how many pods each one is running. It's balancing nodes. You care about pods.

Three nodes pass the health check. Node A runs one pod, node B runs one, node C runs four. The load balancer sends a third of the traffic to each node.

Node A's single pod gets 33% of all traffic. Each of node C's four pods gets about 8%.

A four-to-one spread across identical replicas, from a scheduler decision nobody made deliberately. It shows up as some pods pinned at high CPU while others idle, an HPA that scales on average utilisation and therefore scales at the wrong time, and p99 that tracks whichever pod drew the short straw.

Nothing is broken. Every component is doing its job. The mismatch is that one layer balances nodes and the layer you care about is pods.

Making the distribution match

Two shapes work, and both come down to making pods-per-node uniform.

A DaemonSet gives you exactly one pod per eligible node by construction, which makes node-balancing and pod-balancing the same thing. This is why ingress controllers are so often deployed as DaemonSets, and it's the cleanest answer if the workload is a proxy tier you want on every node anyway.

Otherwise, topologySpreadConstraints with maxSkew: 1 over kubernetes.io/hostname:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: web

DoNotSchedule rather than ScheduleAnyway matters here. The soft version lets the scheduler pile pods onto one node when things get tight, which is precisely the state you're trying to prevent.

Both of these have a cost the docs are quiet about: you've coupled your replica count to your node count. Scaling to 7 replicas across 3 nodes gives you 3/2/2 and a 50% spread that no constraint can remove.

The feature that is not the answer

Kubernetes has grown a newer field that comes up in every discussion of this, and it solves a different problem.

spec.trafficDistribution takes PreferSameZone or, since 1.35, PreferSameNode. The older PreferClose is still accepted and is now explicitly deprecated in the source in favour of PreferSameZone, which means the same thing with a name that says what it does.

It's tempting to read PreferSameNode as a gentler Local. It isn't, for two reasons.

It doesn't preserve the client address. trafficDistribution influences which endpoint kube-proxy picks. It has nothing to say about SNAT, and for NodePort and LoadBalancer traffic the client address still gets rewritten. If source IP is what you came for, this field does not deliver it.

And it's a hint. The API comment is unambiguous: implementations "can use this field as a hint, but are not required to guarantee strict adherence." Local is a rule with a defined failure mode, which is what you want from something load-bearing.

What trafficDistribution does help with is cross-zone data transfer cost and the latency of an extra network hop. Worth having. Different problem.

The limits worth knowing

There's a cheaper answer that people skip past because it feels like a workaround. If your traffic is HTTP and it comes through a layer-7 load balancer or an ingress controller, the client address is already in X-Forwarded-For, and reading it there costs you nothing in distribution. Local earns its keep when you're terminating TCP or TLS yourself, or when you need the address for something below HTTP.

Some clouds also expose a middle path. AWS NLB and the ALB controller can target pod IPs directly rather than NodePorts, which sidesteps the node-level hop entirely and gets you both properties at once. If you're on EKS, check that before reaching for Local.

And a detail worth knowing before you debug this at 3am: pods on a node with Local set can reach the Service through the ClusterIP normally. The policy governs traffic entering from outside. Testing from inside the cluster will not reproduce any of the behaviour described here.

Hyperthreading: the throughput win that costs you latency

Ilia Gusev — Wed, 12 Aug 2026 14:02:09 GMT

Turning hyperthreading off halves your logical core count. Every capacity model you have says that's a bad trade.

Netflix did it on an m7i.metal-24xl and container launch latency improved by 20 to 30%.

The workload was the mount storm from this week's issue: a hundred containers coming up at once, twenty thousand mount syscalls, all funnelling through one global kernel lock. Under that specific shape of load, more logical cores made things worse.

What a hyperthread actually is

An SMT sibling isn't a core. It's a second set of architectural registers bolted onto one physical core, sharing that core's execution units, its L1 and L2 caches, its TLB, and its share of memory bandwidth.

The bet SMT makes is that a single thread leaves a lot of that machinery idle. Waiting on memory, mispredicting a branch, stalling on a dependency. A second thread fills those gaps, and on a mixed workload it usually does, which is where the familiar 20-30% throughput number comes from.

That bet is about throughput. Nobody ever claimed a hyperthread makes an individual thread finish sooner. It can only make it finish later, because the core it was using alone is now shared.

Most of the time you don't care. Then you hit a lock.

Why contention inverts the trade

Take a spinlock, or any lock whose waiters burn CPU rather than sleeping. Two threads on the same physical core, both spinning on the same cache line.

They're now competing twice. Once for the lock, which is the competition you knew about. And once for the execution units and cache of the core they share, which is the one nobody models.

The waiter isn't idle while it spins. It's issuing loads, hammering the same cache line, consuming issue slots. Those are exactly the resources the lock holder needs to finish its critical section and release. So the waiter slows down the holder, which lengthens the critical section, which means more waiting, which means more interference.

Turn SMT off and every runnable thread gets a whole core. Fewer threads spinning at once, and the one holding the lock gets undivided execution resources to get out of the critical section. Total wait drops even though you halved the core count.

The number of logical cores went down. The rate at which the lock changes hands went up. Under contention that second number is the one that decides your latency.

Where this shows up

The pattern needs two things: a hot lock, and enough parallelism to keep both siblings of a core busy fighting over it. That combination is more common than it sounds.

Kubernetes nodes under container churn. CI runners, serverless backends, anything doing rapid pod turnover, where the kernel's mount and cgroup paths become the contended resource. This is the Netflix case.

JVM applications with heavily contended synchronized blocks, or a ConcurrentHashMap where the access pattern collapses onto one bucket. The JVM's biased-lock optimisations stopped helping years ago and were removed in JDK 15.

Databases under high concurrency on one hot table. Postgres buffer-mapping locks, MySQL latch contention. There's a reason database vendors have historically recommended physical-core sizing for these workloads rather than counting vCPUs.

In-memory stores with a global structure. Redis is single-threaded for command execution so it dodges this, but the surrounding I/O threads and anything doing global key-space operations do not.

How to tell whether it's you

The tell is that your tail latency degrades faster than your average as concurrency rises. If p50 is flat and p99 is climbing while CPU utilisation still looks reasonable, contention is a good hypothesis.

From there, perf will tell you where the time goes:

perf lock record -a -- sleep 10
perf lock report --sort wait_total

If a single lock dominates wait_total, you have the first of the two ingredients. Then check whether the busy threads are landing on sibling pairs:

lscpu -e=CPU,CORE,SOCKET | head
# CPU 0 and CPU 24 sharing CORE 0 means those two are siblings

Testing it is cheap and reversible, which is the best thing about this whole topic. You don't need a maintenance window or a new instance type:

echo off | sudo tee /sys/devices/system/cpu/smt/control
# and back
echo on  | sudo tee /sys/devices/system/cpu/smt/control

Run your benchmark both ways on one node. If the answer is no, you've spent ten minutes.

The limits worth knowing

This is not an argument that SMT is bad. For CPU-bound stateless work with little shared state, the throughput win is real and you should take it. Batch processing, video encoding, most stateless request handling that isn't fighting over anything.

The trade also moves with core count. On a machine with a small number of physical cores, halving them hurts more than the contention does, and the inversion may never show up. Netflix measured this on a 24xlarge, where there were plenty of physical cores to go around.

And there's a second reason this button exists. After Spectre, L1TF and the MDS family, several clouds already recommend SMT off for multi-tenant workloads, because sibling threads sharing a core also share the microarchitectural state those attacks read. If you're turning it off for isolation anyway, measure the performance side while you're there. It may not be the cost you budgeted for.

Issue #030 - Mount storm: 20,200 syscalls to start 100 pods

Ilia Gusev — Tue, 11 Aug 2026 14:00:57 GMT

The formula is the whole story, so here it is before anything else:

100 containers x 2 x (1 + 50 + 50) = 20,200 mount operations

That's what Netflix measured on the startup path of a hundred containers with fifty image layers each, after moving from Docker to containerd. Twenty thousand mount syscalls, every one of them taking a global kernel lock, to launch a hundred pods that between them don't mount a single volume.

Nobody wrote that. It emerged from three reasonable decisions - use user namespaces, use overlayfs, use containerd - each defensible, and the multiplication between them nowhere in any of the three designs. That combination is what makes it worth an issue: not a bug in any component, an interaction that only exists at density and only on a node packed the way real nodes get packed.

🏗️ Architectural Pattern: where 20,200 mounts come from

The security upgrade that changed the arithmetic

Old model: a shared host UID range for every container. When a layer was unpacked, file ownership got shifted once, at untar time, on disk. Cheap, done, never thought about again. Also weak - a container escape landed you in a UID space shared with every other container on the box.

New model: each container gets its own host UID range. A break-out from one lands nowhere useful, because the identity space doesn't overlap with anything. This is a genuine improvement and it's the direction everything is going.

The kernel implements it with idmapped mounts. Rather than rewriting ownership on disk, you create a mount of the layer that presents a shifted view of ownership, per container. Per container is the load-bearing phrase. Two containers sharing an image layer can't share its idmapped view, because the whole point is that their UID ranges differ.

So for each layer, per container, the runtime performs the modern mount dance:

open_tree()       # get a detached handle on the layer directory
mount_setattr()   # attach the UID/GID mapping to that handle
move_mount()      # put it where the overlay expects it

Fifty layers, fifty of those. Then one overlayfs mount stacking them. Then, on teardown, fifty unmounts.

And containerd walks that path twice. Once to read user information out of the image so it knows what UID to run as, once to build the actual rootfs. Which gives 2 x (1 + 50 + 50) = 202 mount operations per container, and a hundred containers coming up together makes 20,200.

The lock underneath

Mount operations aren't independent. Linux keeps the mount namespace in a global structure guarded by global locks, and every mount, unmount, and move_mount takes them. A hundred containers starting in parallel don't get a hundred parallel mount paths. They get one, with ninety-nine waiting.

The symptoms Netflix saw are the recognisable shape of this. Reading /proc/mounts taking thirty seconds and more, because generating that file walks the mount table while everyone else is mutating it. systemd falling behind processing mount events. kubelet timing out on containerd, because containerd was blocked in the kernel rather than doing anything wrong.

None of those look like "too many mounts" in a dashboard. They look like a control plane that has gone soft, and the natural response - restart kubelet, drain the node - makes it worse, because a drain is a large batch of unmounts followed by a large batch of mounts somewhere else.

🆚 The Showdown: the hardware fix vs the arithmetic fix

Two ways out, and the comparison is more interesting than either one alone, because they're the same two options you always get with lock contention.

Make the lock cheaper. Global lock throughput depends on how fast cores can pass a cache line between them, and on a modern server that's a function of the interconnect. A monolithic mesh die and a chiplet design with cross-die hops behave very differently under heavy contention on one line, and the gap is large enough that Netflix routed demanding workloads toward the architectures that degrade more gracefully.

It works. It also has an unsatisfying property: nothing got faster, the same 20,200 operations still happen, you just bought hardware that suffers less. The bottleneck is unchanged and the next density increase finds it again.

Do less work. The upstream direction is to stop mounting per layer. Map a common parent directory once, with the container's mapping, and let overlayfs address all the layers underneath it through that single idmapped view. One mount per container instead of one per layer, so the per-container cost drops from O(layers) to O(1), and the hundred-container number drops by roughly two orders of magnitude.

That's the real fix, and it's the one that keeps working when someone doubles the pod density or ships an image with ninety layers.

The comparison generalises past mounts, which is why I keep coming back to it. Kernel global lock plus linear work per unit of workload gives you a scaling limit that's completely invisible until you cross it, and then arrives all at once. Postgres had this with ProcArrayLock and connection counts, fixed in 14 by making snapshots scale rather than by asking people to buy better CPUs. The apiserver has a version of it in watch fan-out under write storms. The pattern is worth carrying around: when you profile a scale problem and find one lock, ask what's linear in front of it before you go shopping.

And a practical corollary. Microbenchmarks won't show you any of this. One container starting on a laptop performs the same 202 mounts and never contends, because there's nobody to contend with. Density limits have to be tested at density, on the CPU topology you actually run.

👮 The Policy: layer count is a startup-latency metric

The reader-actionable part is small and nobody does it: count the layers in your images.

crane config  | jq '.rootfs.diff_ids | length'

Or across everything running:

kubectl get pods -A -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' \
  | sort -u

then run the first command over that list. Most people are surprised twice - once by the maximum, once by how many images sit above forty when the base image alone accounts for a dozen.

Layer count has been treated as an image-size concern for a decade, and on that metric it barely matters, because layers dedupe and registries cache. On mount cost it's linear and it doesn't dedupe, because idmapped views are per container by construction. That reframes the usual image-hygiene advice: multi-stage builds, collapsing RUN chains, and distroless bases aren't only about pull time and CVE surface. They are also about how many times the kernel takes a global lock when your node reboots.

Worth putting a check in CI. A build that adds five layers to a base image is fine; one that ships sixty because every apt-get got its own line is a node-density problem waiting for a busy Monday.

The honest part

This is a density problem and most clusters aren't dense. At thirty pods a node with twenty-layer images you're looking at roughly 1,200 mount operations spread over a rolling start, and you'll never see it. The teams that hit this are running a hundred-plus pods per node, or restarting a whole node's worth of workloads at once, or both.

The other honest bit: you probably can't apply the real fix yourself. Whether the per-layer dance collapses into one mount is decided by your container runtime and kernel version, not by anything in your manifests. What you control is the multiplier. Fewer layers, and not restarting every pod on a node simultaneously when a rolling drain would do.

If you want to know whether it's you, mountsnoop from bcc-tools on a node during a scale-up will tell you in a minute. Count the syscalls and compare with the formula. If the number matches, you now know exactly which term to attack.

The class of bug this belongs to

What I like about this one is that no individual decision was wrong. User namespaces per container is correct. Idmapped mounts are the right implementation. Overlayfs stacking layers is how images work. Containerd reading image config before building a rootfs is reasonable. Multiply them and you get twenty thousand syscalls through a single lock.

That's most interesting infrastructure failure, in my experience. Not a bad component, four good ones whose costs compose in a direction nobody was measuring, showing up only past a density threshold that the people who designed each piece never tested at.

I went and counted the layers in our own images after reading the Netflix writeup, expecting to feel smug. Our worst one is at forty-three. It's a Python service, and thirty of those layers are a base image that somebody assembled carefully, one dependency per layer, to get good cache behaviour in CI. Which was a completely sound decision, made for a completely different metric.

Questions? Feedback? Reply to this email. I actually read them.

Ilia

Instance-store NVMe: the fastest disk that erases itself

Ilia Gusev — Fri, 07 Aug 2026 14:01:41 GMT

Somebody resized the node group. Bigger instance type, same everything else, a routine change done during business hours because it carries no risk.

Changing an EC2 instance type means stop, modify, start. And a stop wipes every instance-store volume attached to that host, permanently, before the instance comes back. The ClickHouse data directory was on one of them.

The cluster recovered from replicas and nobody lost a customer. What stayed with me is that nothing in the change ticket, the Terraform plan, or the console warned about it. The disk's called a volume, it mounts like a volume, df reports it like a volume. It just isn't one.

Reboot keeps it, stop shreds it

The rule has an edge that catches people, because two operations that feel equivalent are not:

Reboot: data survives
Stop, then start: gone
Hibernate: gone
Terminate: gone
Underlying host failure: gone

Reboot keeps the instance on the same physical host, so the disks are still right there. Stop releases the host. When you start again you get a different machine with different disks, and AWS will not carry the old ones across.

The wording in the AWS documentation is worth reading closely, because it describes a mechanism rather than a policy: when the instance is stopped, hibernated, or terminated, every block of the instance-store volume is cryptographically erased. There's no dangling copy that a support ticket can recover. The key is destroyed and the blocks become noise.

Hibernate is the one that catches people who read half the docs. Hibernation saves RAM to the root EBS volume and feels like a pause rather than a stop, so the instinct is that everything survives. The instance-store volumes stay attached and their contents are gone.

Why it's fast, in one sentence

The disk is on the PCIe bus of the host your instance runs on. EBS is storage over a network, however good that network is.

That's the whole difference, and everything else follows from it. No network hop means latency in microseconds rather than milliseconds, and IOPS in the hundreds of thousands to millions rather than the tens of thousands an EBS volume will give you without a provisioned-IOPS bill. You also stop paying for the IO, because there isn't anything to meter: the disk comes with the instance.

The trade is not performance versus cost. It's performance against whether the disk exists tomorrow.

The taxonomy that actually decides it

One question sorts almost every workload: if this disk vanished right now, who reconstructs the data, and how long does it take?

If the answer is "nobody, it's gone" then instance store is wrong, whatever the benchmark says.

If the answer is "the system does it, automatically, and we've tested that" then instance store is often the best storage in EC2 for the job. Three shapes qualify.

Caches. Redis or Valkey or Memcached holding hot data that has a source of truth elsewhere. Losing it costs you a cold period, not data.

Scratch space. ETL intermediates, video encoding, model training checkpoints you can regenerate, build artifacts. Anything where the file exists to be consumed and then deleted.

Distributed stores that own their replication. ClickHouse, Cassandra, ScyllaDB, Elasticsearch data nodes. A node that loses its disk gets rebuilt from peers, which these systems are designed around rather than treating it as an incident.

The trap in that third category is the word "distributed". A three-node cluster with replication factor 1 is distributed and it'll still lose data permanently when one node stops. The property that matters is the replication factor, not the topology.

What this means on Kubernetes

Instance store shows up in a cluster as a node-local disk, which puts you in one of two patterns.

Either you use it as ephemeral storage for the kubelet itself, which is a straightforwardly good idea. Container images, emptyDir, and the writable layer all live on /var/lib/kubelet and /var/lib/containerd. Putting those on instance-store NVMe makes image pulls and container startup measurably faster, and everything there is already expected to disappear with the node.

Or you use it for a StatefulSet through a local-volume provisioner, and that's where care is needed. A local PersistentVolume binds a pod to a specific node. When that node goes away the PV becomes unusable and the pod stays Pending until somebody cleans it up. That's the correct behaviour for local storage, and it's nothing like the recovery story people expect from a PVC.

If you're on Karpenter or any consolidation-happy autoscaler, add this to the list: consolidation stops and replaces nodes. That's the same erase path as the manual resize. Workloads on instance store need to either tolerate node replacement by design, or carry a karpenter.sh/do-not-disrupt annotation and an explicit story for how you ever patch them.

The limits worth knowing

You can't attach instance store after the fact. It comes with the instance type or it doesn't, and the only way to add it is to launch a different type. That's the opposite of EBS, and it changes capacity planning: storage becomes a property of the instance family, not a dial.

Storage-optimized families are where it lives in quantity, and both the sizes and the naming move faster than any article can track. Check the current instance-type table rather than trusting a number you read somewhere, including here.

And one operational note that costs people an afternoon. On most storage-optimized types the NVMe devices arrive raw. No filesystem, no mount, no entry in /etc/fstab. A fresh instance has fast empty disks that nothing's using until userdata or your AMI formats and mounts them, which is exactly the kind of step that works on the golden image and gets forgotten in the Terraform module.

NetworkPolicy runs after DNAT: the hairpin nobody tests

Ilia Gusev — Wed, 05 Aug 2026 14:02:26 GMT

The policy was the careful kind. Egress allowed to 0.0.0.0/0 on 443, with the RFC1918 ranges carved out, so pods could reach the internet but not wander around the private network. It passed review. It worked in staging for three weeks.

In production it dropped a call that had nothing to do with private networks: a pod talking to the company's own API over its public hostname.

The manifest was identical in both places. What differed was where the packet got rewritten.

The bit the manifest doesn't tell you

A NetworkPolicy is written against addresses. Packets get matched against addresses. In between sits the part nobody diagrams: kube-proxy, or your CNI's replacement for it, rewriting the destination.

Pod sends to 203.0.113.10:443, the load balancer address for api.example.com. That address resolves back into the cluster, so the packet never leaves. A Service picks it up and DNATs the destination to a backend pod, 10.244.3.17:8443.

Now the policy gets evaluated. So what address does it see?

If enforcement happens before DNAT, it sees 203.0.113.10 and the rule matches 0.0.0.0/0. Allowed.

If enforcement happens after DNAT, it sees 10.244.3.17, which lands squarely in the 10.0.0.0/8 block you carved out. Dropped.

Same manifest. Same intent. Opposite outcome, decided by a mechanism the manifest can't see.

The spec says it's undefined

This is where it stops being a CNI bug and starts being a design gap. From the Kubernetes NetworkPolicy documentation:

> Cluster ingress and egress mechanisms often require rewriting the source or destination IP of packets. In cases where this happens, it is not defined whether this happens before or after NetworkPolicy processing, and the behavior may be different for different combinations of network plugin, cloud provider, Service implementation, etc.

And more directly:

> Connections from pods to Service IPs that get rewritten to cluster-external IPs may or may not be subject to ipBlock-based policies.

"May or may not" is doing a lot of work in a security primitive. Nobody's going to fix this in a patch release, because there is nothing to fix. Two implementations can both be conformant and disagree.

GKE takes the honest route and refuses the question: in Dataplane V2 you can't put a Pod or Service IP in ipBlock.cidr at all. The API rejects it rather than accepting a rule whose meaning depends on packet-rewrite ordering.

Why ipBlock is the fragile selector

Every other NetworkPolicy selector matches on something Kubernetes owns. podSelector matches labels. namespaceSelector matches labels. Those are stable facts in etcd, and no datapath component rewrites them mid-flight.

ipBlock matches an IP address, which is the one field in the packet that half your infrastructure exists to change. kube-proxy rewrites it. The CNI rewrites it. The cloud load balancer rewrites it. Your egress gateway rewrites it. The rule's written against the one value that's explicitly in motion.

Which gives a rule of thumb worth more than any specific workaround: use label selectors for anything inside the cluster, and treat `ipBlock` as a tool for addresses that live outside the cluster. If the target has a Pod or a Service in front of it, select the Pod.

The carve-out policy above should have been two rules. Allow egress to the API by podSelector plus namespaceSelector, since it is in the cluster. Allow egress to 0.0.0.0/0 on 443 with private ranges excluded for everything that really is outside. Once the in-cluster case is handled by labels, the DNAT ordering stops mattering, because no rewritten address is ever matched against a CIDR.

Where the dev/prod split comes from

The reason this passes staging is that hairpin routing is not universal.

In a small cluster, a pod reaching for the external hostname of a Service in the same cluster often gets hairpinned by the node: the packet turns around at the local datapath and gets DNATed there. In a production setup with a real cloud load balancer, the same request can leave the node, reach the LB, and come back with a rewrite performed somewhere else entirely.

Two paths, two rewrite points, one policy. The environment where you test the policy is the environment where its behaviour is defined, and only there.

Testing that catches it looks like this: from a pod in the restricted namespace, curl the target the way the application actually addresses it. Not the Service DNS name if the app uses the public hostname. Not the public hostname if the app uses the Service. The address the app uses is the only one whose rewrite path you're testing.

kubectl -n restricted run probe --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' --max-time 5 https://api.example.com

%{remote_ip} is the useful part. It tells you which address curl actually connected to, which tells you whether you're on the hairpin path or the external one before you start reasoning about policy.

The limits worth knowing

Knowing your CNI's enforcement point helps, but it isn't a general answer. Cilium enforces in eBPF at the socket and tc layers, Calico in iptables or eBPF depending on the mode, and both move that boundary between versions. Anything you learn today is a fact about the version you're on, not about the API.

There's one thing the spec does guarantee, and it surprises people the other way: traffic to and from the node a pod runs on is always allowed, whatever your rules say. A policy that looks like it blocks the node isn't blocking the node.

And the honest summary of the whole area: NetworkPolicy is a good tool for expressing which workloads may talk to which workloads. It's a poor tool for expressing which IP ranges a workload may reach, because IP ranges aren't what it's built on. When you find yourself reaching for ipBlock to describe something inside your own cluster, that's usually the signal to go back to labels.

Podo Stack

Bitnami, one year later: hardened base images went free

September 29, 2025: the tags disappeared

Links

December 17, 2025: Docker gave its images away

Links

2026: everyone priced against zero

Links

What "near-zero CVEs" counts

Links

What teams did with their charts

Links

The recipe was never the expensive part

subPath: the ConfigMap update 7 of our 12 pods never saw

Five pods had crashed, and those five got the new value

Links

What the kubelet writes into a ConfigMap volume

Links

A CVE lived in the same resolve-then-mount step

Links

What we changed

The honest part

Links

Where it landed

Issue #035 - HolmesGPT, Job retries, and the IfNotPresent hole from 2015

🚀 Sandbox Watch: HolmesGPT

What it is

How it works

The honest part

Where I'd use it

Links

💎 The Hidden Gem: backoffLimitPerIndex

What it is

One budget for a thousand shards

What per-index changes

Where it still bites

Links

⚔️ The Showdown: Only one of these tenancy tools changes your namespace count

The setup

How each one draws the line

The #023 cost angle

My read

Links

👮 The Policy: IfNotPresent let pod B run an image it had no right to pull

Pod B never showed a credential

What the fixes cost

The manifests

What stays open

Links

The rolling update myth: your frontend is talking to next week's backend

Why this is worse in full-stack frameworks

What's actually going on: version skew

Links

Doing it on plain Kubernetes

The trade-off that makes this non-trivial

When to reach for this

The scheduler that starts 7 of your 8 GPU pods, then bills you for all 8

Why this isn't a podAffinity problem

Gang scheduling: the actual fix

Links

How Kueue actually enforces it

Links

Topology awareness: gang scheduling isn't enough on its own

This isn't only an ML problem

Issue #034 - Gonzo, CloudEvents, and the container that held while the policy didn't

🚀 Sandbox Watch: Gonzo

What it is

How it works

The honest part

Links

💎 The Hidden Gem: CloudEvents

What it is

The envelope

Who actually uses it

Where it stops helping

Links

⚔️ The Showdown: Lens went paid, Kubernetes Dashboard went dark, one project answers both

The setup

The one everyone still recommends, and shouldn't

What's actually alive

Why this isn't a `podAffinity` problem