Podo Stack

The autoscaler that hid a prefix cache collapse: 80% more GPUs

Ilia Gusev — Fri, 25 Sep 2026 14:03:30 GMT

In February Nick Roan published a short post-mortem on Medium about a pull request that broke nothing. His team changed RAG chunking from 512 tokens to 500 to fit a downstream context budget. Answers stayed accurate and the unit tests passed. The prefix cache hit rate went from about 85% to about 4%, and in his words: "On the same traffic volume, GPU replicas increased 80% while p95 latency remained within SLO."

Those numbers are his. The post doesn't name the autoscaler or its scaling signal and gives no timeline after the merge, so the dashboard below is a reconstruction.

The graph with nothing red on it

Illustrative values shaped to his numbers, two GPUs per replica, flat traffic:

$ ./slo-cost-report.sh --around merge --step 12h
# illustrative, shaped to the public write-up: hit rate 85% -> 4%, replicas +80%
# SLO: ttft_p95 < 1s
window   prefix_hit  ttft_p95  replicas  req_s  gpu_s_per_1k_req
T-48h    0.86        0.41s     10        31.8   629
T-24h    0.85        0.43s     10        32.4   617
T-12h    0.85        0.40s     10        31.5   635
T+0h     0.31        0.97s     11        32.0   688
T+12h    0.05        0.63s     16        31.9   1003
T+24h    0.04        0.55s     18        32.2   1118
T+48h    0.04        0.56s     18        31.6   1139

TTFT spikes for one window and settles under the line. An HPA does what its formula says: desired replicas equal current replicas times current metric over target, rounded up. GKE's guide for LLM inference recommends queue size or batch size as that metric, and a queue grows the same way whether traffic doubled or every request got several times more expensive to prefill. Replicas climbed until the queue drained. Two columns stayed up after that, replicas and GPU-seconds per thousand requests, and neither one is a latency SLO.

Why twelve tokens stranded everything behind them

vLLM keeps the KV cache in 16-token blocks by default and caches only full ones. Per the prefix caching design doc, each block's hash covers its own token IDs plus the hash of the block before it, so one hash fingerprints the whole prompt up to that point. The full-attention lookup in vllm/v1/core/single_type_kv_cache_manager.py walks the chain from the start and breaks on the first miss. A comment above that loop reads: "A missing block implies every later block misses too (chained hashes)."

Roan's post walks the blocks. Everything through token 495 matched, 31 blocks of it. Block 31 held the first token the new chunking changed, and each block after it carried text the cache had seen before under a parent hash it had never seen, so none of them hit.

SGLang's RadixAttention keeps the cache in a radix tree keyed by token sequences, with LRU eviction, instead of a table of hashed blocks. It is still a prefix structure, and a divergence near token 500 orphans everything beneath that node the same way. A date string at the top of a system prompt does it from block zero.

The honest part: a boundary shift alone should heal. I copied vLLM's block hashing and first-miss lookup into a Python replay and fed it 3,000 RAG prompts over 100 document-topic groups. Right after the switch to 500 the token hit rate fell to 53% for 100 requests, then ran at 97% for the remaining 2,900 while new prefixes warmed. A rate that stays near 4% needs a prefix that changes on nearly every request. The post doesn't say what did that. My guess is retrieval re-run every turn and placed ahead of the conversation history.

What the alert looks at now

Roan's fix lives in CI. An ephemeral vLLM container on a T4 or A10G runner replays about 50 multi-turn conversations twice, warm-up then audit, and fails the build when hit rate drops or recomputed tokens rise against a baseline from main. Tokenizer and model revisions are pinned, and hashing runs with sha256_cbor, since the default sha256 serializes through pickle and may not reproduce across Python or vLLM versions.

One detail the post gets loose: it calls the hit rate "request-level", but vllm/v1/metrics/stats.py counts both queries and hits in tokens. If his formula reads those counters, 85% meant 85% of prompt tokens came from cache. The old V0 gauge vllm:gpu_prefix_cache_hit_rate is gone, and the V1 counters dropped their gpu_ prefix in v0.10.0. On v0.29.0, released September 9, three queries cover it:

# token-weighted prefix cache hit rate
sum(rate(vllm:prefix_cache_hits_total[15m]))
  / sum(rate(vllm:prefix_cache_queries_total[15m]))

# prompt tokens computed per request, cache hits excluded (v0.13+)
sum(rate(vllm:request_prefill_kv_computed_tokens_sum[15m]))
  / sum(rate(vllm:request_prefill_kv_computed_tokens_count[15m]))

# GPU-seconds per 1k requests
1000 * sum(
  kube_pod_container_resource_requests{namespace="llm", resource="nvidia_com_gpu"}
  * on (namespace, pod) group_left () (kube_pod_status_phase{phase="Running"} == 1)
) / sum(rate(vllm:request_success_total{namespace="llm"}[15m]))

The middle one also catches a prompt that doubled in length at a steady hit rate, the case Roan's second gate was built for. Nothing in the GPU-seconds query cares how many replicas the autoscaler considers fine.

More replicas, fewer hits

The cache lives in each replica's GPU memory unless a KV connector shares it, so going from 10 pods to 18 behind a plain Service, as in the table above, means every hot prefix gets prefilled once per pod before it hits. In the same toy replay, 300 requests against cold pods reached 74% on one pod, 58% on two, 42% on four and 31% on eight. Steady state closes most of that gap when KV memory is big enough, but scale-out pods start cold and show up while the fleet is already short.

llm-d measured the real version with Qwen3-32B on 8 vLLM servers across 16 H100s, using 150 prefix groups that each open with a 6,000-token system prompt. Behind a stock Service TTFT p90 reached 135.5 s at the top of the rate ladder, while the llm-d router held it under 0.3 s and pushed 2.1x the output throughput. The endpoint picker, which moved from the Gateway API Inference Extension repo to llm-d/llm-d-router, scores pods from an approximate index of prefixes it already routed or from KV-cache events vLLM publishes over ZMQ.

Routing only reunites a request with a prefix that still exists. After Roan's merge, a perfect picker would have found a warm pod holding 496 matching tokens and nothing past them.

One integer

CDN teams know this from the other end: a tracking parameter slips into the cache key and the origin pool scales out for traffic that never grew. Client-side sharding over Redis does it when a new node remaps keys and the miss rate reads as load to everything behind it.

The pull request in Roan's story changed 512 to 500.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

API priority and fairness: 600 seats, a 22-object list took 10

Ilia Gusev — Wed, 23 Sep 2026 14:04:37 GMT

Every five minutes our tenant-operator logs filled up with "the server has received too many requests and has asked us to try again later". Argo CD refreshes that normally took two seconds stretched to eleven. I ran kubectl get pods -n platform against the same cluster (1.36, three apiserver replicas) in the middle of one of those bursts and it came back in under a second. Apiserver CPU sat at 38%, etcd p99 at 9 ms.

The operator reconciles 40,000 TenantBinding objects spread over 1,800 tenant namespaces. A few weeks earlier someone had added a drift sweep: every five minutes, one goroutine per namespace, each doing an uncached LIST through controller-runtime's API reader. Client QPS had been raised to 1000 for a load test and never lowered. On every five-minute mark about 1,800 LISTs landed within a second or two.

Where the 600 seats go

kube-apiserver budgets concurrency in seats. Each replica gets --max-requests-inflight (400) plus --max-mutating-requests-inflight (200), so 600 seats per replica, and API Priority and Fairness divides them between priority levels in proportion to nominalConcurrencyShares. The objects shipped in 1.36 and 1.37 add up to 245 shares:

workload-low: 100 shares, 245 seats
workload-high, node-high: 40 shares, 98 seats each
system: 30 shares, 74 seats
global-default: 20 shares, 49 seats
leader-election and catch-all: 25 and 13 seats; exempt bypasses the count

A FlowSchema decides which level a request lands in. Our operator runs as a service account in the platform namespace, so it matched the service-accounts FlowSchema (matchingPrecedence 9000) and went to workload-low, next to Argo CD and every other service account outside kube-system. My kubectl identity is a human user. It fell through to global-default at 9900 and never touched workload-low's seats.

workload-low also ships with lendablePercent 90. While it's quiet, other levels can borrow 221 of its 245 seats, leaving a floor of 24, and the borrowing controller only re-divides every 10 seconds.

The dump

The apiserver serves its APF state, per replica, on an unlisted debug path. I trimmed the columns:

$ kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels \
    | grep -E 'Name|global|workload' | cut -d, -f1,5,6,8,10 | sed 's/Requests//g' | column -t
PriorityLevelName,  Waiting,  Executing,  Rejected,  Cancelled
global-default,     0,        2,          0,         0
workload-high,      0,        11,         0,         0
workload-low,       71,       24,         18406,     212

$ kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1' \
    | awk -F', *' '/tenant-operator/ {print $1, $7, $12, $14}' | head -2
workload-low 10 list tenant-a0412
workload-low 10 list tenant-b0077

$ kubectl get tenantbindings -n tenant-a0412 --no-headers | wc -l
      22

Twenty-four requests executing in a level with 245 seats. The second column of that awk output is InitialSeats: a LIST returning 22 objects was holding 10 seats. One oddity if you read this dump yourself: the TimedoutRequests column is never incremented anywhere in the current queueset.go, so queue time-outs show up as Rejected plus Cancelled.

Why 22 objects cost 10 seats

Since 1.34 the SizeBasedListCostEstimate feature gate (beta, on by default) prices a LIST at one seat per 100 KB the apiserver expects to hold in memory. The estimate takes the object count and average size of the whole resource, whatever namespace the URL names. 40,000 TenantBindings at about 9 KB each is 360 MB. When the estimator assumes the watch cache serves the LIST, it caps memory at 1 MB, so the charge tops out at 10 seats, and every namespace-scoped LIST we sent paid that maximum.

LISTs it expects to hit etcd, a continue-token page for example, skip the 1 MB cap and stop at a per-level limit instead: the smaller of 15% of nominal seats and nominal seats divided by handSize. In workload-low that comes to 37. The kubernetes.io page still describes seats as proportional to object count, and a comment in apf_controller.go says the estimator "limits max seats at 10", while request/config.go raised the LIST ceiling to 100 in July 2025.

Seat width also stalls dispatch. The dispatcher takes the head of the queue with the smallest virtual finish time, and if that head needs 10 seats while 6 are free, nothing dispatches. All 1,800 LISTs were one flow (service-accounts distinguishes by user), shuffle-sharded onto 6 of 128 queues with 50 slots each. Roughly 1,400 were rejected on arrival. The rest could wait a quarter of their deadline, 15 seconds under the default 60-second timeout. Retry-After starts at 1 second and doubles while drops continue, up to 32, and client-go retries a 429 up to 10 times, so one sweep kept coming back for minutes.

Argo CD hashed to other queues and never got a 429. It waited behind 10-seat heads, which is where the eleven-second refreshes came from.

What we changed

The sweep moved to the manager's cached client that same afternoon. It reads TenantBindings from the informer the operator already had, so the five-minute burst stopped reaching the apiserver. QPS went back to defaults.

Then we fenced the operator's LISTs into their own level, so the next uncached read hits a smaller wall:

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: tenant-sweep
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 20
    lendablePercent: 50
    borrowingLimitPercent: 0
    limitResponse:
      type: Queue
      queuing:
        queues: 8
        handSize: 2
        queueLengthLimit: 50
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: tenant-operator-lists
spec:
  priorityLevelConfiguration:
    name: tenant-sweep
  matchingPrecedence: 1000
  distinguisherMethod:
    type: ByUser
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount:
            name: tenant-operator
            namespace: platform
      resourceRules:
        - verbs: ["list"]
          apiGroups: ["platform.example.com"]
          resources: ["tenantbindings"]
          namespaces: ["*"]
          clusterScope: true

With 20 more shares the total becomes 265: tenant-sweep gets 46 seats per replica and workload-low drops to 227. Leaving borrowingLimitPercent out would have meant unlimited borrowing, which defeats the fence. The per-request cap follows the level size too, so a LIST in tenant-sweep is charged at most 7 seats. The operator's writes and watches still match service-accounts. We also added alerts on apiserver_flowcontrol_rejected_requests_total by priority_level and reason, and on p99 of apiserver_flowcontrol_request_wait_duration_seconds.

Twenty-three days later

Rejections in workload-low have stayed at zero since the change. tenant-sweep sees a few dozen LISTs a day from a startup check that still uses the API reader, and none of them has queued. The drift sweep finishes in 1.9 seconds from memory, and Argo CD refreshes at :00 and :05 look like the ones at :03. The first fix proposed in the incident channel was raising --max-requests-inflight to 600. That would have handed workload-low 82 more seats, against a sweep that asked for 1,800 requests at 10 seats each every five minutes.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

Issue #036 - PodGroup, skopeo, and the RBAC verb that skips escalation checks

Ilia Gusev — Tue, 22 Sep 2026 14:03:52 GMT

This week I mirrored registry.k8s.io/pause:3.10 into a local registry with skopeo. The plain copy tagged a single platform and the upstream index digest was gone. Adding --all brought the index back under a different digest, because skopeo gzipped two uncompressed Windows layers on the way out. The digest only survived with --preserve-digests, a flag that is off unless you pass it.

The other three blocks turned on defaults too. Grab bag, round four.

📑 The KEP Read: Gang scheduling is in kube-scheduler now, switched off

What changed in v1.37

On September 9 we published a Kueue piece that credited the default scheduler with placing pods "one at a time, as capacity allows." Kubernetes 1.37 had shipped gang scheduling inside kube-scheduler as beta 14 days earlier, after an alpha in 1.35. I missed it.

KEP-4671 promoted Workload and PodGroup to scheduling.k8s.io/v1beta1, and the old GangScheduling and WorkloadAwarePreemption gates were folded into GenericWorkload. Shared DRA claims for a PodGroup went beta too, and CompositePodGroup (KEP-6012) arrived as alpha for groups of groups.

Beta does not mean enabled. GenericWorkload defaults to false, and v1beta1 is on the apiserver's disabled-by-default list. I started a v1.37.0 kube-apiserver locally. With only the gate, podgroups wasn't served; with only the runtime-config flag, the log said "PodGroup storage is disabled". It took both, plus the gate on kube-scheduler and kube-controller-manager.

How a gang gets placed now

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: trainer
spec:
  schedulingPolicy:
    gang:
      minCount: 8
  disruptionMode:
    all: {}
---
apiVersion: v1
kind: Pod
metadata:
  name: trainer-0
spec:
  schedulingGroup:
    podGroupName: trainer
  restartPolicy: Never
  containers:
  - name: worker
    image: registry.example.com/trainer:2.4
    resources:
      limits:
        nvidia.com/gpu: 1

minCount is the only gang knob and has been mutable since 1.37. There is no timeout field. The scheduler holds member pods until at least minCount of them exist, then evaluates the group against one cluster snapshot. When 7 of 8 fit, none bind. Workload-aware preemption gets a try, then the pods return to the unschedulable queue and PodGroupInitiallyScheduled turns False with reason Unschedulable.

What it doesn't do and where Kueue still sits

The KEP lists fairness and multiple queues as a non-goal: "Kueue and Volcano.sh will continue to provide this." Quota and admission stay in Kueue. So does waitForPodsReady, since the scheduler never updates a group's status after placement, even when pods crash or get evicted.

The only built-in controller that creates PodGroups is Job, behind the alpha WorkloadWithJob gate. Kueue support for the in-tree PodGroup is an open umbrella issue, #8871, with nothing shipped as of v0.20.0-rc.0.

The honest part

Beta still churns. In 1.37 spec.podGroupTemplateRef became spec.workloadRef in types.go (PR #140080), though the PodGroup concept page still shows the old name. The changelog also says to delete v1alpha2 objects before upgrading from 1.36.

Managed clusters are mostly locked out. GKE Rapid has had 1.37 since September 2, but GKE documents switching off-by-default beta gates only for alpha clusters, which get deleted after 30 days. EKS standard support ends at 1.36. AKS release notes from September 4 announce a 1.37 preview rollout and say nothing about workload-aware scheduling.

So the Kueue piece needs a qualifier: "one at a time" still holds on every production managed cluster and on any 1.37 cluster where nobody set the gate. It also quoted Kueue v0.13.4, a September 2025 release. The current one is v0.19.4.

💎 The Hidden Gem: pvc-autoresizer

What it is

A small controller from the TopoLVM project (the LVM CSI driver that started in Cybozu's cybozu-go GitHub org) that edits PVCs when their filesystems run low. It never calls a cloud API: it raises the storage request and leaves the expansion to the CSI driver. Apache-2.0, 411 stars, v0.21.1 out on September 2, Kubernetes 1.33 through 1.35 in the README.

How it decides

Each loop (1 minute in the binary, 10 seconds in the Helm chart) it asks Prometheus for kubelet_volume_stats_available_bytes and its capacity twin, plus free and total inodes. When free space or free inodes drop under their thresholds, the request grows by the increase, rounded up to a whole GiB. Both default to 10%. The README says a percentage increase is based on the spec request; internal/runners/pvc_autoresizer.go uses status.capacity.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-autoresize
  annotations:
    resize.topolvm.io/enabled: "true"      # opt-in per StorageClass
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true                 # admission rejects PVC resizes without it
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-ingest-0
  annotations:
    resize.topolvm.io/storage_limit: 2Ti   # required: no limit, PVC skipped without a log line
    resize.topolvm.io/threshold: 15%       # of filesystem capacity (default 10%)
    resize.topolvm.io/increase: 20%        # of status.capacity (default 10%)
spec:
  accessModes: [ReadWriteOnce]
  volumeMode: Filesystem                   # Block volumes are ignored
  storageClassName: gp3-autoresize
  resources:
    requests:
      storage: 1Ti

Where it waits

AWS dropped the six-hour EBS cooldown on January 15. A volume now gets four modifications in a rolling 24 hours, each after the previous one reaches completed. The extra space appears within seconds, yet AWS's docs say a 1 TiB volume can take up to six hours to finish. When we replayed a log flood against a 1 TiB gp3 claim, the first resize landed in under a minute. Forty minutes later the second bounced off the EBS CSI driver with "in OPTIMIZING state, cannot currently modify", and the disk hit 100% while external-resizer retried.

pvc-autoresizer doesn't stack requests meanwhile. It stores the filesystem size in resize.topolvm.io/pre_capacity_bytes and waits for kubelet to report a new one (a maintainer walked through this in issue #296). A resize that never completes leaves it logging "waiting for resizing..." for good.

Scrape gaps pass unnoticed: a stale kubelet series drops the PVC from the query, and pvcautoresizer_failed_resize_total stays flat.

GCP runs its own clocks, two size changes per 4 hours on Hyperdisk Balanced and one per 6 on Extreme PD. No major cloud shrinks a disk, and Kubernetes rejects a request below status.capacity, so storage_limit is what a runaway log gets to spend and keep.

Nothing native

Kubernetes has nothing native. VolumeAttributesClass (GA in 1.34) changes IOPS or type, and on EBS it spends the same four-a-day budget. I found no KEP for volume autoscaling, and DevOps-Nirvana's Kubernetes-Volume-Autoscaler last released in October 2023. I'd leave the Metrics API mode off: the README points at metrics-server, but the code reads each kubelet through nodes/proxy, the permission from issue #033.

⚔️ The Showdown: Mirroring the pause image with skopeo changed its digest twice

The setup

registry.k8s.io/pause:3.10 is a seven-entry index, five Linux images and two Windows ones, digest ee6521f2. I mirrored it into a local registry:3 and asked each copy for that digest.

skopeo 1.24.0 went first. A plain copy runs with --multi-arch=system, and on a linux/arm64 box that meant one child: the mirror's tag pointed at e50b7059, and a pull by ee6521f2 got MANIFEST_UNKNOWN. An amd64 run lands on 7c38f247 instead.

--all brought every platform back and still missed. The Windows images ship uncompressed layers, and docker/docker_image_dest.go in go.podman.io/image tells a registry destination to compress. One 265 MB layer went out as 109 MB of gzip, and the index landed at 37fae201. The man page doesn't mention it. --preserve-digests returned ee6521f2, and so did a laptop rerun whose blob cache had already seen the raw layers.

Indexes and referrers

crane 0.22.1 and regctl 0.11.6 pushed the index byte for byte; cmp found nothing. Each drops to one platform only when handed a platform flag.

For referrers I used regctl's own image. Its referrers flag carried 33 manifests: a cosign bundle on the index, plus two SBOMs, a cosign bundle and a BuildKit attestation on each of eight platform images. registry:3 returned 404 on the Referrers API, which zot from #033 serves natively, so they landed as nine sha256-digest fallback tags. The digest-tags flag handles the older .sig tags pause:3.10 uses. The crane and skopeo copies had none: cmd/crane has no referrers path, and skopeo moves .sig attachments only when use-sigstore-attachments is set in registries.d.

Sync and day-2

skopeo sync takes a repo or a YAML list with tag filters, keeps copy's defaults and exits after one pass.

regclient's regsync runs as a server: interval or cron per entry, a ratelimit block that waits out source pull limits, a backup tag before an overwrite, the same referrers and digestTags switches. crane has no sync; gcrane cp -r walks repositories through Google's listing API.

One caveat on regclient: it is nearly a one-person project. Brandon Mitchell, who maintains the OCI distribution and image specs, has 1,792 commits; the next contributor has two. 1,930 stars; skopeo, part of a CNCF Sandbox project, has 11,231.

My read

Mirror that must keep digests, signatures and SBOMs

regctl with referrers and digest tags; regsync once it runs on a timer.

Promotion step in a pipeline that already has crane

crane copy. Index intact, referrers left behind.

Existing skopeo jobs

--all --preserve-digests on every copy and sync.

Anything pinned by index digest

crane digest on both sides before the rollout.

👮 The Policy: RBAC already blocks escalation, unless you hand out escalate

The check that already runs

On a v1.37.0 test apiserver alice held a ClusterRole that creates Roles and RoleBindings. Her Role granting get and list on Secrets came back with: user "alice" (groups=["dev-team" "system:authenticated"]) is attempting to grant RBAC permissions not currently held.

An archive note behind this block claimed a team lead could do exactly that and blocked it with a Kyverno validate.roles rule, a field Kyverno doesn't have.

That refusal lives in pkg/registry/rbac/role/policybased/storage.go, with a twin for bindings, and runs before admission. Its "you already hold it" half reads RBAC objects only, which the docs never mention. On GKE that means IAM-granted permissions don't count.

Three verbs and a group that turn it off

escalate on roles or clusterroles skips the check on their contents, aggregationRule guard included. Our ci-deployer service account, holding escalate and bind in one namespace, wrote a secret-reader Role, bound it to itself, and can-i on secrets flipped from no to yes.
bind on a role lets you bind it without holding what it grants.
impersonate borrows an identity instead. Built-in edit carries it for serviceaccounts, and admin inherits it. Constrained impersonation (KEP-5284, default-on beta since v1.36) adds verbs like impersonate:serviceaccount that a literal match misses.
system:authenticated as a subject. The check never reads the subject list: alice bound her own role to that group, and bob could create ClusterRoles a second later. On GKE that means any Google account; Orca's January 2024 "Sys:All" scan flagged 1,300 of 250,000 clusters.

rbac-tool who-can escalate clusterroles
kubectl auth can-i --list -n payments --as=system:serviceaccount:payments:ci-deployer

The manifest

Swap platform-admins for your group. On the same apiserver it denied every case above, * verbs on the RBAC group and an aggregationRule, and let platform-admins through. Without the aggregation-controller exemption, edit and admin stop picking up aggregated rules, and edit already contains impersonate.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: rbac-escalation-guard
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["rbac.authorization.k8s.io"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
  matchConditions:
  - name: not-platform  # masters = apiserver bootstrap, the SA = admin/edit/view aggregation
    expression: >-
      !request.userInfo.groups.exists(g, g in ["system:masters", "platform-admins"]) &&
      request.userInfo.username !=
        "system:serviceaccount:kube-system:clusterrole-aggregation-controller"
  variables:
  - name: riskyRules
    expression: >-
      (has(object.rules) && object.rules != null ? object.rules : []).filter(r,
        r.verbs.exists(v, v in ["escalate", "bind"] || v.startsWith("impersonate")) ||
        ("*" in r.verbs && has(r.apiGroups) && has(r.resources) &&
          r.apiGroups.exists(g, g in ["*", "", "rbac.authorization.k8s.io"]) &&
          r.resources.exists(x, x in ["*", "roles", "clusterroles",
                                      "users", "groups", "serviceaccounts"])))
  - name: publicSubjects
    expression: >-
      (has(object.subjects) ? object.subjects : []).map(s, s.name).filter(n,
        n in ["system:authenticated", "system:unauthenticated", "system:anonymous"])
  validations:
  - expression: size(variables.riskyRules) == 0
    reason: Forbidden
    messageExpression: >-
      "escalate/bind/impersonate/wildcard verbs are platform-only: " +
      variables.riskyRules.map(r, r.verbs.join(",")).join("; ")
  - expression: "!has(object.aggregationRule) || object.aggregationRule == null"
    reason: Forbidden
    message: aggregationRule is platform-only
  - expression: size(variables.publicSubjects) == 0
    reason: Forbidden
    messageExpression: '"binding to " + variables.publicSubjects.join(", ") + " is blocked"'
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: rbac-escalation-guard
spec:
  policyName: rbac-escalation-guard
  validationActions: ["Deny"]

What it doesn't stop

New grants are all it stops. An identity already holding escalate still writes any Role - ci-deployer's secret-reader went through with the policy on, so the who-can list matters more than the YAML.

Crossplane v2.4.0's RBAC manager holds escalate and bind, and Argo CD v3.5.3's application-controller ships with * on everything, so a synced chart carrying a Role with bind gets denied. Their service accounts go into matchConditions by name, as does the CI identity running Helm.

Exempting system:masters keeps apiserver bootstrap working and doubles as break-glass; #026 has the zero-logins rule for that path.

GKE 1.28+ refuses cluster-admin bindings to the three public subjects; every other role still binds. Two opt-in gcloud flags (1.30.1-gke.1283000+) block the rest and leave existing bindings alone.

Bitnami, one year later: hardened base images went free

Ilia Gusev — Fri, 18 Sep 2026 14:03:50 GMT

This week I ran helm show values on the Apache Airflow chart, version 1.22.0, released in June. The bundled PostgreSQL points at bitnamilegacy/postgresql, tag 16.1.0-debian-11-r15. The image labels say it was built on November 30, 2023, on Debian 11, and Debian 11 left LTS on August 31.

That image is there on purpose. A year ago Bitnami shut most of its free catalog and said running it cost too much. Eleven weeks later Docker gave its hardened images away under Apache 2.0, and by August 2026 a startup that matched that price had announced its shutdown.

September 29, 2025: the tags disappeared

Broadcom posted the notice on July 16, 2025, in bitnami/containers issue #83267. On August 28 Bitnami stopped publishing Debian-based images and OCI charts to Docker Hub, then ran brownouts through September.

The cut came on September 29. That morning someone filed an issue because docker.io/bitnami/kafka had no tags at all. The maintainer replied that Kafka wasn't in the trial set: about 40 images and 10 charts stayed free, latest tag only, for development. Versioned tags were frozen in docker.io/bitnamilegacy. Production moved to Bitnami Secure Images, rebuilt on VMware's Photon OS, with no public price; press coverage put it at $50,000 a year and up.

The reasoning came in a Broadcom community post on August 18: "Bitnami has been the Jenkins of the internet for many years, but this has become unsustainable. Operating a build pipeline and OCI registry for the general public is very expensive."

AWS dropped its 317 Bitnami repositories from ECR Public on June 10, 2026. Today docker.io/bitnami/postgresql has no version tags, only latest, which is PostgreSQL 18.6 on Photon OS 5.0:

$ docker pull bitnami/postgresql:16.4.0
Error response from daemon: manifest for bitnami/postgresql:16.4.0 not found: manifest unknown: manifest unknown

December 17, 2025: Docker gave its images away

Docker launched Docker Hardened Images on May 19, 2025, as a paid product with a 7-day patch window for critical and high CVEs. When Bitnami users went shopping that August, Docker called DHI "affordable and transparent", which still meant a trial and a sales conversation.

On December 17 more than 1,000 images and Helm charts went free under Apache 2.0, served from dhi.io. A Docker account is the entry ticket; an anonymous token request to dhi.io comes back Unauthorized. Unlike Bitnami's leftovers, the free tier keeps versioned tags such as python:3.13. Select and Enterprise hold the SLA and the FIPS and STIG variants, and Enterprise alone gets five years of Extended Lifecycle Support past upstream end of life. The public definitions repo held 595 image definitions and 96 charts when I counted this week.

One developer quoted by DevClass wasn't sold: "It's free for now, just like registries were 'free' and docker desktop was free... until they weren't."

2026: everyone priced against zero

Chainguard had the most to lose and moved the least. Its free images at cgr.dev/chainguard need no login but ship as latest and latest-dev only, a set it trimmed in November 2024. Anonymously, python:latest resolves for me and python:3.12 returns 404. In December CEO Dan Lorenc told TechTarget he'd be "pretty cautious" about Docker's offer. On March 17, 2026, Chainguard launched Catalog Starter: five images of your choice, all supported versions, production use allowed, no CVE SLA, corporate email required.

Red Hat followed on May 12 with Red Hat Hardened Images, a no-cost catalog of more than 45 images.

Minimus went furthest. On June 23 it opened its whole catalog, every major and minor version, no registration. By late August its homepage said the reg.mini.dev registry switches off on October 22. Echo bought the technology on August 27 and told TechTarget it would keep the platform running. Chris Hughes of Noma Security told the same outlet: "Everyone kind of raced to 'out-free' each other."

What "near-zero CVEs" counts

Every catalog above advertises near-zero CVEs, so I went looking for the denominator. Docker's number is what a scanner reports after applying Docker's own VEX statements. A CVE Docker judges unexploitable gets not_affected with a justification, and a backported fix without a version bump gets not_affected plus inline_mitigations_already_exist. Docker's docs warn that scanners without VEX support report more. The Grype example in those same docs, VEX applied, still shows a High perl CVE marked won't fix.

Chainguard publishes its triage as security advisories in wolfi-dev/advisories and secdb feeds that scanners consume. Its docs say plainly that a CVE count depends on build age, and show an October 2024 jre digest scanning dirty. The paid SLA clock starts once a qualifying patch exists: seven days for critical, fourteen for everything else.

The honest reading of zero is a fresh build scanned by a tool that accepts the vendor's triage. Pin that digest for six months and you run a six-month-old image with the same badge.

What teams did with their charts

On September 18, 2025, mid-brownout, Airflow merged "Temporary fix to Bitnami psql chart licensing issues", which swapped in the official postgres:13 image, a major-version downgrade from 16. In January 2026 a follow-up put back the exact Bitnami image, now from the archive: "It's likely users of the provided postgres moved to this image anyways." Mastodon pointed its Bitnami subchart images at bitnamilegacy in October 2025. Superset did the same on May 4, 2026, "until we can switch over to Valkey and vanilla Postgres."

GitLab split it. The company's infrastructure team parked its Redis caches on bitnamilegacy/redis as a short-term fix, while the product chart removed bundled Bitnami PostgreSQL and Redis in GitLab 19.0 on May 21, 2026, "with no replacement", and the migration guide points at CloudNativePG and Valkey. Langfuse's chart 2.0.0, out August 18, replaced Bitnami Redis with the official Valkey chart and Bitnami Postgres with groundhog2k/postgres. The Valkey issue covered where that chart came from.

The advice from both ends of the removal was mirroring. A Bitnami maintainer told one user to copy an image "to your own registry before the final removal." AWS, retiring its own copies, wrote: "Never point production workloads directly at a public registry you don't own." A mirror with digest pins trades a tag that vanishes for a digest that never changes, and someone still has to move the pin.

The recipe was never the expensive part

Bitnami's Dockerfiles still sit in bitnami/containers under Apache 2.0, with commits from this week. Anyone can build them. Broadcom's complaint was about running "a build pipeline and OCI registry for the general public", and the vendors doing that for free in 2026 have something else to sell next to it. Docker has subscriptions and Red Hat has RHEL; Minimus had only the images.

Until somebody opens the next PR, Airflow's chart will keep pulling a Postgres build from November 2023.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

subPath: the ConfigMap update 7 of our 12 pods never saw

Ilia Gusev — Wed, 16 Sep 2026 14:03:40 GMT

On Tuesday, September 1, we raised one number in the checkout-api ConfigMap. upstream_timeout went from 2s to 5s, because a payment provider's p99 had crept up to 3.1 seconds. Twelve replicas. The service has had hot reload since 2024, so nobody restarted anything. Argo CD showed Synced, and the ConfigMap in the cluster said 5s.

By Wednesday morning timeout errors were down from about 410 an hour to about 240. I took that as the provider still being slow and moved on.

On Friday a node pool upgrade drained every node the service ran on. All twelve pods came back elsewhere, and timeout errors fell to 9 an hour. Three days of a config change that had "rolled out" and mostly hadn't.

Five pods had crashed, and those five got the new value

On Monday I split the error graph per pod. From Tuesday night on, seven pods produced every timeout and five produced none. Those five all had a restart count of 1 or 2, OOMKilled on Tuesday night by a memory limit set too tight months earlier. The seven that never restarted had been reading 2s all along.

Then I opened the chart. It mounted the ConfigMap with subPath: app.yaml at /etc/checkout/app.yaml, because the image ships other files in /etc/checkout that a directory mount would hide. The docs cover this in one note: "A container using a ConfigMap as a subPath volume mount will not receive ConfigMap updates." I'd read that page before. It hadn't stuck.

What the kubelet writes into a ConfigMap volume

I reproduced it in staging: a throwaway pod mounting the same ConfigMap as a directory, next to a checkout-api pod with the subPath mount. I bumped the timeout from 5s to 7s. 71 seconds later the probe pod saw 7s, and the app pod still read 5s.

$ kubectl -n stg exec cfg-probe -- ls -la /cfg
total 12
drwxrwxrwx 3 root root 4096 Sep  7 10:14 .
drwxr-xr-x 1 root root 4096 Sep  7 10:12 ..
drwxr-xr-x 2 root root 4096 Sep  7 10:14 ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root   32 Sep  7 10:14 ..data -> ..2026_09_07_10_14_31.3920571846
lrwxrwxrwx 1 root root   15 Sep  7 10:12 app.yaml -> ..data/app.yaml

Nothing in that directory is a plain file. app.yaml links to ..data/app.yaml, and ..data links to a timestamped directory holding the real bytes. On an update, the kubelet's AtomicWriter writes the new payload into a fresh timestamped directory, points a ..data_tmp symlink at it and renames that over ..data, so a reader never catches half a file. Then the old directory gets deleted.

A subPath mount never looks at ..data again. At container start the kubelet runs the path through filepath.EvalSymlinks, lands on the real file inside the timestamped directory and bind-mounts that one file into the container. The swap moves ..data later, and the bind mount keeps holding the old file - deleted from the volume, still readable through the mount. A container restart makes the kubelet resolve the path again, see a different inode and remount (PR #89629, in since v1.19), which is why our OOMKilled pods were the lucky ones. Env vars from a ConfigMap are also read once, at start.

Our 71 seconds fit the documented formula: kubelet sync period plus cache propagation delay. Defaults are Watch for configMapAndSecretChangeDetectionStrategy and 1m for syncFrequency, and the pod worker requeues each pod with a 0.5 jitter factor, so a running pod gets re-synced roughly every 60 to 90 seconds.

A CVE lived in the same resolve-then-mount step

I only found this one in the issue tracker while reading up on subpath_linux.go. CVE-2021-25741 was a symlink exchange in that code path: a container's subPath mounts could reach files outside the volume, host filesystem included, which gave hostPath-like access on clusters that had banned hostPath. It was rated High and fixed in v1.22.2, v1.21.5, v1.20.11 and v1.19.15, the second subPath symlink CVE after CVE-2017-1002101. Today's kubelet opens the subpath safely and bind-mounts /proc//fd/ instead of a path the container could swap.

What we changed

checkout-api lost its subPath. The ConfigMap now mounts as a directory at /etc/checkout/conf.d/, with the config flag pointing there. Its hot reload is viper's WatchConfig, which watches the parent directory and re-checks the resolved symlink target on every event (the source comment names "k8s ConfigMap replacement"). Three edits in staging gave three reloads.

The ledger service was worse off. Its hand-rolled reload put an fsnotify watch on the file path, and inotify follows symlinks, so the watch sat on the real file inside the timestamped directory. The first swap deleted that directory and the watch with it, and no second update ever arrived. We moved the watch to the directory and re-read on any event touching ..data, which is what the AtomicWriter comment suggests.

Services with no reload got a checksum/config annotation on the pod template, so a ConfigMap edit changes the template and the Deployment rolls normally. Our two third-party charts went under Stakater's Reloader (10.4k stars, v1.4.22 on September 9) with reloader.stakater.com/auto set to "true".

ConfigMaps from Kustomize's configMapGenerator already carry a content-hash suffix, so those got immutable: true. The docs sell it on performance: watches on immutable objects get closed, which counts at tens of thousands of ConfigMap-to-pod mounts (more on what watches cost the apiserver). We're nowhere near that. I wanted the next in-place edit to fail at apply time, loudly.

The honest part

Hot reload through a directory mount skips the rollout. When I pushed a bad 50ms timeout on purpose to staging, all four replicas were serving with it 80 seconds later. A checksum rollout would have gone pod by pod under maxUnavailable. That route is slower and noisier, and I trust it more for anything that can break request handling.

Where it landed

On September 10 we changed upstream_timeout again. The per-pod error graph moved as one line this time, and all twelve pods logged the reload within 88 seconds of the Argo CD sync.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

Issue #035 - HolmesGPT, Job retries, and the IfNotPresent hole from 2015

Ilia Gusev — Tue, 15 Sep 2026 14:03:08 GMT

Capsule's API description says a tenant quota's hard limit "is never crossed." The controller behind it sums usage on a reconcile loop and rewrites each namespace's limit to whatever headroom is left, so between two passes every namespace sees the same leftover. I only found that because I stopped at the sentence and opened the code.

It happened again in all four blocks this week. Grab bag, round three.

🚀 Sandbox Watch: HolmesGPT

What it is

An open-source agent for incident investigation. You ask "why is ledger crashlooping?", it queries your cluster and observability stack, then writes up a root cause. Robusta started it in May 2024 and donated it to the CNCF, where the Sandbox vote passed on October 8, 2025. The repo moved from robusta-dev to its own HolmesGPT org. Apache 2.0, about 3,350 stars, release 0.41.0 on September 8.

How it works

An agentic loop, where the model reads each tool result before choosing its next call. Built-in toolsets reach Kubernetes, Prometheus, Loki, Tempo, Datadog, ArgoCD and dozens more, plus MCP servers. Runbooks are now called skills, markdown procedures synced from git. Models go through LiteLLM, with OpenAI as the default and Anthropic, Bedrock or Ollama a flag away. It ships as a CLI, a Helm-deployed HTTP API, a k9s plugin, or a Slack bot via Robusta.

The honest part

"By design, HolmesGPT has read-only access," says the README. I went looking for where that gets enforced. The Kubernetes toolset skips secrets, and the Helm chart's ClusterRole grants none. A bash toolset ships enabled too, though, with "kubectl get" in its core allowlist and an empty default deny list. On the CLI it runs under your kubeconfig, so a secret read passes whenever your RBAC permits it. Anything off the list stops at a "Do you want to proceed?" prompt, which --bash-always-allow removes. Since February a Remediation MCP server can restart, scale, drain and patch, each change human-approved. That's the policy question from issue #034, inside an SRE tool.

Every log line the loop pulls lands in the model's context, off your network with a hosted provider. Ollama support is labeled experimental, with tool calling that "may produce inconsistent results."

Accuracy, from the project's own August 5 benchmark: seven models passed between 71% and 94% of 63 runs, and five scored 50% on tests tagged "hard". A wrong answer prints under the same "AI:" header as a right one.

Where I'd use it

k8sgpt, Sandbox since December 2023, finds problems with Go analyzers and calls an LLM only for --explain. HolmesGPT lets the model choose what to query, so it reaches further and guesses more. I'd take it for the first ten minutes at 3 a.m., read its conclusion as a list of places to look, and keep --bash-always-allow away from any admin kubeconfig.

💎 The Hidden Gem: backoffLimitPerIndex

What it is

A retry budget per index for Indexed Jobs. backoffLimitPerIndex and maxFailedIndexes went GA in Kubernetes 1.33 and have been on by default since 1.29. Around them the Job API kept growing: podFailurePolicy reached GA in 1.31, successPolicy in 1.33, podReplacementPolicy in 1.34, managedBy in 1.35. The retry knob most people remember is still the original one, backoffLimit, default 6.

One budget for a thousand shards

Picture a 1000-shard Indexed Job with that setting. The controller compares the 6 against status.failed for the whole Job, a counter that doesn't reset when other shards succeed, and fails the Job on the seventh failed pod from any index. Three crashes on shard 7 plus four pods caught in a node drain will do it. Running pods get deleted, unstarted indexes never get a pod, and nothing retries the Job afterwards.

Drains count immediately. Without a pod failure policy, a pod is treated as failed the moment it gets a deletionTimestamp, even if it would have exited 0 inside its grace period.

The docs say retry backoff (10s, doubling) caps at six minutes. I checked pkg/controller/job, and the cap constant has been 10 * time.Minute since 1.28.

What per-index changes

Each index gets its own failure counter and its own backoff clock. A shard that burns through its budget lands in status.failedIndexes while the other 999 keep going. Once failed indexes go past maxFailedIndexes, the controller terminates the Job. Under that cap it still ends Failed, reason FailedIndexes, but with 997 shards done and three index numbers to rerun.

An Ignore rule on the DisruptionTarget pod condition keeps evictions and preemptions off the count, and FailIndex gives up on an index at once on an exit code reserved for bad input.

apiVersion: batch/v1
kind: Job
metadata:
  name: reindex-shards
spec:
  completionMode: Indexed
  completions: 1000
  parallelism: 50
  # no backoffLimit here: with backoffLimitPerIndex set it defaults to 2147483647
  backoffLimitPerIndex: 2      # third failed pod marks this index failed
  maxFailedIndexes: 20         # 21st failed index terminates the Job
  podFailurePolicy:
    rules:
    - action: Ignore           # drain, preemption, taint eviction, node pressure
      onPodConditions:
      - type: DisruptionTarget
    - action: FailIndex        # bad input for this shard, retrying won't help
      onExitCodes:
        containerName: worker
        operator: In
        values: [42]
  template:
    spec:
      restartPolicy: Never     # required by both backoffLimitPerIndex and podFailurePolicy
      containers:
      - name: worker
        image: registry.example.com/reindex:1.8.2
        args: ["--shard=$(JOB_COMPLETION_INDEX)"]

Where it still bites

Leftover backoffLimit in the manifest still wins. The controller checks the Job-wide limit first, so a chart that templates backoffLimit: 6 onto every Job brings the shared budget back.

Ignoring DisruptionTarget also swallows kubelet node-pressure evictions. If a shard pushes its node into memory pressure, it gets rescheduled with no retry limit unless activeDeadlineSeconds is set. An OOM kill against the container's own memory limit still counts.

kubectl describe job prints Completed Indexes and nothing for failed ones; those live only in status.failedIndexes.

Version lag is small. The oldest EKS release still in extended support is 1.31, and upstream every field above is on by default there. managedBy is the exception, alpha and off until 1.32.

⚔️ The Showdown: Only one of these tenancy tools changes your namespace count

The setup

I went to pull the Hierarchical Namespace Controller into this comparison and found a read-only repo under kubernetes-retired. SIG Auth voted in February 2025 to archive it "due to a lack of maintainers and adopters".

How each one draws the line

Accurate copies objects down a tree. Annotate a Secret or Role with accurate.cybozu.com/propagate and it lands in the sub-namespaces below. Budgets are out of scope. The sample config even lists ResourceQuota as a propagatable kind, and a copied quota gives each child the full limit. v1.9.0 in June was mostly Renovate bumps. 79 stars.

Capsule is the governance layer: a Tenant owns a capped set of namespaces and pins allowed registries and storage classes. It has sat in CNCF Sandbox since December 2022 and has 2,175 stars; v0.14.5 shipped last Friday. The tenant quota's API description promises the hard limit "is never crossed", so I read the controller. The legacy spec.resourceQuotas is a reconcile loop that sums usage and rewrites each namespace's hard to the leftover headroom. Between passes, every namespace sees the same leftover. v0.14.0 (August 21) added GlobalResourceQuota, which reserves usage atomically at admission, and the old quota system is now deprecated.

With vCluster, the tenant stops being a namespace. Each one gets its own API server, OSS under Apache 2.0. Embedded etcd and Private Nodes need the Free tier, which validates its license against vCluster Platform. v0.37.1 landed yesterday, 11,298 stars.

The #023 cost angle

In #023 the bill scaled with namespace count. Capsule and Accurate leave that count where it was, and Accurate inflates the object count on top: one Secret propagated over 40 sub-namespaces is 41 Secrets in every cluster-wide Secret informer.

vCluster moves the line. Tenant namespaces and CRDs stay in the virtual control plane; the syncer pushes pods, services, endpoints and PVCs, plus only the ConfigMaps and Secrets pods reference, into one host namespace per tenant.

The honest part: pods still run on host nodes, so pod-watching DaemonSets see every one of them. And the bill moves into control planes. The chart default asks for 200m CPU and 256Mi, but the sizing guide recommends 4 CPU and 8 GiB per replica, three replicas, for production. Fifty tenants at that profile is 600 CPU in limits.

My read

Trusted teams sharing Secrets and RBAC

Accurate. Opt-in copying, nothing more.

Internal platform with real per-team budgets

Capsule 0.14+, shared limits in GlobalResourceQuota.

Namespace count already hurting, or tenants need their own CRDs

vCluster, after pricing the control planes.

Still on HNC

Archived since April 2025. The archive thread points at Accurate.

👮 The Policy: IfNotPresent let pod B run an image it had no right to pull

Pod B never showed a credential

Pod A in team-a runs registry.example.com/private/app:1.4.2 with an imagePullSecret. Later the scheduler drops pod B from team-b onto the same node - same image, no imagePullSecrets, default IfNotPresent. The kubelet asks the runtime whether the image is on disk, hears yes, and logs "Container image ... already present on machine". No registry call, so no credential check. kubernetes/kubernetes#18787 reported this in December 2015 and closed in April 2025.

The archive note behind this block forced Always through a Kyverno ClusterPolicy and promised "no limitations." Neither half holds up in 2026.

What the fixes cost

AlwaysPullImages. An admission plugin, off by default, that sets Always on every Pod and rejects anything else. For an unchanged image the pull is a manifest check measured in kilobytes, per KEP-2535. The bill is availability: the kubelet re-checks on every container start, CrashLoopBackOff restarts included, so a registry outage stops pods whose bytes already sit on the node.

KEP-2535, inside the kubelet. The KubeletEnsureSecretPulledImages gate went alpha in v1.33 and flipped to beta, on by default, in v1.35. v1.37 still ships it as beta. Every kubelet pull leaves a record under /var/lib/kubelet/image_manager/ with the secret's namespace, name, UID and credential hash, kept across restarts. Pod B brings no matching credential, so the kubelet pulls for real and the private registry refuses it. A pod reusing a known secret skips the registry.

The policy kind is on its way out too: Kyverno 1.19 deprecated ClusterPolicy on August 20, and 1.20 removes it.

The manifests

For 1.35+ nodes whose kubelet config you own:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# NeverVerify | NeverVerifyPreloadedImages (default)
# NeverVerifyAllowlistedImages | AlwaysVerify
imagePullCredentialsVerificationPolicy: NeverVerifyAllowlistedImages
preloadedImagesVerificationAllowlist:
  - registry.k8s.io/*   # public only: allowlisted images skip verification entirely

For nodes you can't configure, or older than 1.35, force Always for the private prefix and nothing else. MutatingAdmissionPolicy went GA in v1.36, so there's no webhook to run:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
  name: private-images-always-pull
spec:
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE"]
      resources: ["pods"]
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >
        Object{
          spec: Object.spec{
            containers: object.spec.containers
              .filter(c, c.image.startsWith("registry.example.com/"))
              .map(c, Object.spec.containers{name: c.name, imagePullPolicy: "Always"})
          }
        }
  - patchType: ApplyConfiguration
    applyConfiguration:
      expression: >
        Object{
          spec: Object.spec{
            initContainers: object.spec.?initContainers.orValue([])
              .filter(c, c.image.startsWith("registry.example.com/"))
              .map(c, Object.spec.initContainers{name: c.name, imagePullPolicy: "Always"})
          }
        }
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicyBinding
metadata:
  name: private-images-always-pull
spec:
  policyName: private-images-always-pull

I ran it against a v1.37.0 kube-apiserver: both private containers came back Always, and an nginx:1.27 sidecar kept IfNotPresent. Kyverno 1.18+ takes the same mutations block in a policies.kyverno.io/v1 MutatingPolicy.

What stays open

Under the default policy, any image the kubelet didn't pull itself counts as preloaded and stays open to every pod on the node. That includes crictl-based warmers like the DaemonSet in #020, and everything already on disk when the gate first turned on. An in-place upgrade from 1.34 leaves a node full of those. AlwaysVerify closes the gap and puts the registry back in front of them.

GKE's node system config has no field for this policy; EKS AL2023 nodes take raw KubeletConfiguration through nodeadm. Records don't expire either: revoke a registry token and pods referencing that same secret keep starting the cached image until image GC removes it.

Links

Issue #18787 went up on GitHub in December 2015 and was closed in April 2025. The default that shuts the gap shipped with v1.35 that December, ten years after the report. It covers images the kubelet pulled itself, so a crictl preloader like the one in #020 still leaves pod B a way in.

Subscribe now

Questions? Feedback? Reply to this email. I actually read them.

Ilia

The rolling update myth: your frontend is talking to next week's backend

Ilia Gusev — Fri, 11 Sep 2026 14:02:38 GMT

A user loads your app at 2:47pm. Their browser caches the JavaScript bundle - React Server Component payloads, API client code, the works. At 2:52pm, your rolling update finishes: new backend pods are live, old ones are gone. The user, still on the same tab, clicks something. Their cached frontend, built for deployment N, calls an API shaped for deployment N+1.

Sometimes that's a 500. Sometimes it's a silent RSC hydration failure. Other times it's a contract mismatch that only shows up as "weird" data three clicks later. Nobody on call connects it to the deploy, because the deploy finished five minutes ago and "succeeded."

Rolling updates are safe for the backend. They say nothing about whether the client talking to that backend is the version you think it is.

Why this is worse in full-stack frameworks

Any API can drift out of sync with an old client for a few seconds during a rollout - that's not new. What's new is how tightly coupled frontend and backend have become in frameworks like Next.js, Remix, SvelteKit, and Nuxt: shared types, server components whose payload shape is the contract, code that assumes the two sides were built together because, until a rolling update happens, they were.

And the window isn't seconds anymore. A user's session - and the JS bundle their browser is holding onto - can outlive your rolling update by minutes, hours, sometimes days, depending on caching and how long they leave the tab open. "The rollout only takes two minutes" doesn't help if the client that matters loaded ten minutes before it started.

What's actually going on: version skew

This has a name - version skew - and once you know to look for it, the fix is controlling which backend version a given client session actually talks to, for as long as that session lives, not shipping rollouts faster.

Vercel shipped this first, for Next.js specifically: Skew Protection. A deployment ID gets attached to the client, carried via a cookie (__vdpl), and every subsequent request from that session gets routed back to the exact deployment that originally served it - not whatever's currently live. The session stays pinned to one version until its own natural end, regardless of how many deploys happen underneath it.

Doing it on plain Kubernetes

Vercel's version is a managed platform feature. The concept ports to vanilla Kubernetes without needing that platform, using pieces you likely already have: a session cookie that carries a deployment identifier, Gateway API routing rules that read that cookie and send the request to the matching backend version, and a small state machine tracking each deployment as Active, Draining, or Expired so you know when it's finally safe to tear an old version down.

None of this replaces your rollout strategy - canary, blue/green, whatever you're already doing. Skew protection sits on top of it. The rollout question is "how do I move traffic to the new version safely." The skew question is a separate one: "once a specific user's session started on version N, does it stay on version N until that session naturally ends."

The trade-off that makes this non-trivial

Pinning sessions to versions means old versions have to keep running, potentially for as long as your longest-lived session - which means running N versions of your backend simultaneously instead of one, with memory, replica count, and infrastructure cost scaling with N. Observability gets messier too: every metric now needs a version label, or your dashboards quietly average together three different backends and tell you nothing useful.

It's also not a fit for everything. A clean backend API with no shared types and no client-side caching to speak of mostly doesn't have this problem in the first place - skew protection is solving a specific coupling, not a general Kubernetes deployment concern.

When to reach for this

Anywhere you don't control the client cache - mobile apps that don't force-refresh, CDN edge caching, long-lived SPA sessions - rolling updates alone aren't enough, and the fix looks more like blue/green scoped to the session than a faster rollout. It's the failure mode that falls squarely between the frontend team and the platform team, which is usually why it ships unfixed: each side assumes the other one owns it.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The scheduler that starts 7 of your 8 GPU pods, then bills you for all 8

Ilia Gusev — Wed, 09 Sep 2026 14:02:52 GMT

A distributed training job asks for eight GPU pods. Seven schedule within seconds. The eighth sits in Pending - one node short of capacity somewhere in the cluster - and stays there. Nothing crashes. Nothing errors. The training process on all seven running pods just hangs, because the all-reduce step that synchronizes gradients across the job is waiting for a peer that never showed up.

Seven GPUs, burning money, doing nothing, for however long it takes someone to notice and kill the job.

The default Kubernetes scheduler did exactly what it was built to do: schedule pods, one at a time, as capacity allows. That's precisely the problem.

Why this isn't a `podAffinity` problem

The instinct is to reach for podAffinity or topologySpreadConstraints to keep the job's pods close together. Neither one touches the actual failure. Both operate per-pod, evaluated independently as each pod is scheduled - they can influence where a pod lands, but nothing about them makes eight pods land together or not at all. The scheduler still admits each pod as its own decision, and a job that needs all eight can still end up with seven.

Distributed training, MPI jobs, Spark executors that all need to see each other before work starts - none of these have a "partial success" mode. A job that gets 7 of 8 pods isn't 87% done. It's 0% done, at full cost.

Gang scheduling: the actual fix

Gang scheduling flips the admission decision from per-pod to per-job: either every pod in the group gets scheduled together, or none of them do, and the ones that would've scheduled successfully get held back too. HPC batch schedulers have handled this exact problem for decades; Kubernetes borrowed the concept late, and it shows up now as "all-or-nothing" semantics in a handful of projects: Kueue, Volcano, Apache YuniKorn.

Kueue is the one worth knowing first: an official Kubernetes SIG-Scheduling project (kubernetes-sigs/kueue), not some third-party fork of the scheduler, currently at v0.13.4 and shipping regularly.

How Kueue actually enforces it

Kueue sits in front of the default scheduler as an admission layer, rather than replacing it. Jobs go into a LocalQueue, a namespace-scoped object your team submits work to, which references a ClusterQueue that owns the actual quota and fair-sharing rules across the cluster. Kueue holds a job back until it's confident the whole group can be admitted, then releases all of its pods to the underlying scheduler together.

The enforcement mechanism is waitForPodsReady: a cluster-wide, timeout-based check. Once Kueue admits a workload, it watches until every pod in it reports ready. If the timeout passes before that happens - because a node disappeared, or capacity got contended by something else - Kueue evicts the whole workload and requeues it, rather than leaving it half-running and silently burning resources. That's the mechanism that turns "7 of 8, hanging forever" into "0 of 8, retried automatically."

Topology awareness: gang scheduling isn't enough on its own

Getting all eight pods scheduled somewhere solves the all-or-nothing problem. It doesn't solve the next one: distributed training over NVLink or 400G Ethernet is latency-sensitive enough that where those eight pods land relative to each other changes throughput by a lot, not a little.

Kueue's topology-aware placement understands a zone-over-rack-over-host hierarchy and bin-packs a job's pods onto adjacent nodes rather than scattering them wherever capacity happens to be free. Gang scheduling gets you eight pods running together. Topology awareness gets you eight pods running together fast.

This isn't only an ML problem

The specific example is GPU training because that's where the pain is loudest right now, but the underlying failure mode - a job whose pods have to arrive together to do useful work at all - applies to any tightly-coupled distributed system: MPI jobs, Spark executor pools, anything with a synchronization barrier before real work starts. If your workload has a moment early on where every worker waits for every other worker, the default scheduler's per-pod admission model is working against you, not for you.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

Issue #034 - Gonzo, CloudEvents, and the container that held while the policy didn't

Ilia Gusev — Tue, 08 Sep 2026 14:01:42 GMT

Kubernetes Dashboard is archived. Lens is a subscription. Between those two, one project ended up as the answer to both - which made this week's Showdown block write itself. That's the shape underneath all four of these: something that looked settled - a boundary, a standard, a free tool, a security model - turning out to hold less than advertised.

Grab bag, round two.

🚀 Sandbox Watch: Gonzo

What it is

A terminal log viewer built specifically for OTLP, from ControlTheory, a small observability startup. It surfaced publicly around KubeCon 2025 USA, MIT licensed, sitting around 2,800 GitHub stars now.

How it works

Built on Bubble Tea, Charm's TUI framework - not the tview/tcell stack k9s runs on. "Inspired by k9s" means the 2x2 grid layout and vim-style keybindings, not shared code. It runs its own OTLP receiver - gRPC on 4317, HTTP on 4318 - so logs land directly, no Loki or Elasticsearch sitting in between. It also auto-detects JSON, logfmt, and plain text, with prebuilt integrations for Kubernetes pod logs, Docker, syslog, and a long list of platforms: Vercel, Supabase, Railway, Cloudflare Workers, Netlify, Fly.io, Render, AWS CloudWatch.

The honest part

A Show HN commenter reported 5,000 log lines taking over six minutes to process on their machine - "so slow" was the exact complaint. Worth checking against your own log volume before you commit to it. There's also a fixed bug about returning terminal control to the parent process "(e.g. k9s)" on quit, which tells you people are already running this as a k9s plugin rather than standalone.

And it isn't purely a community project. Gonzo shares a blog and a company with ControlTheory's paid product, an AI-focused observability platform pitched right alongside it. That doesn't make the free tool worse - it's still MIT, still yours to run - but it's worth knowing you're also looking at a funnel.

💎 The Hidden Gem: CloudEvents

What it is

The CNCF's standard envelope for events - not a message bus, not a delivery protocol, just an agreed shape for "here's what happened." Spec v1.0.2, Graduated in the CNCF in January 2024, health score rated "Excellent." Google, Microsoft, IBM, Red Hat, VMware, SAP, and Serverless Inc. all had a hand in it - a vendor consortium, not one inventor.

The envelope

Four required fields: id, source, specversion, type. A handful of optional ones: datacontenttype, dataschema, subject, time. That's genuinely it - small enough to hold in your head, which is the whole point.

Who actually uses it

Knative Eventing is the deepest integration - Broker and Trigger are built directly around CloudEvents. Dapr wraps pub/sub messages in a CloudEvents envelope by default. AWS EventBridge and Azure Event Grid both support it too, but as an opt-in mode through API Destinations or a schema setting - not what you get by default.

Where it stops helping

The spec doesn't touch routing or delivery guarantees - that's left entirely to whatever's actually moving the event around. And dataschema is optional, which means the envelope is standardized but the payload inside it usually isn't. A consumer still can't validate a CloudEvent's actual data without a separate, out-of-band agreement on what that data looks like. CloudEvents solves "what shape is this event," not "what does it mean" or "will it arrive."

⚔️ The Showdown: Lens went paid, Kubernetes Dashboard went dark, one project answers both

The setup

Mirantis has been tightening Lens's licensing since 2023 - marketplace extensions gated, multi-cluster limited, an Enterprise tier added in 2024. Since October 1, 2024, even the free Personal plan lost team features - shared clusters and Spaces now need a Pro or Enterprise subscription.

The one everyone still recommends, and shouldn't

OpenLens - the open-source fork people reach for reflexively - is dead. Last release was v6.5.2 in June 2023. No cease-and-desist, no drama: once Mirantis stopped shipping an open core to fork from, contributions simply stopped. If a "just use OpenLens" comment is more than a year old, don't trust it.

What's actually alive

FreeLens is the real successor - a fresh community fork started in 2024/2025, sitting around 5,300 stars, cross-platform, no proprietary licensing anywhere in it. If you want the original Lens desktop experience without a subscription, this is it now, not OpenLens.

The other half of the story

Kubernetes Dashboard - the official web UI - was archived on January 21, 2026. The repo is read-only. And the project's own farewell note points people toward Headlamp.

Headlamp has been CNCF Sandbox since May 2023, with Microsoft engineers (formerly Kinvolk) actively maintaining it, a monthly release cadence, and both web and desktop builds behind a plugin system. It's the one project that both halves of this story - the paywalled desktop app and the abandoned official dashboard - now point toward.

My read

Terminal-first: k9s, untouched by any of this. Want the original Lens desktop feel for free: FreeLens. Want the option positioned to be the default going forward, with a real team behind it and a plugin ecosystem: Headlamp. Kubernetes Dashboard has dropped out of the running entirely - it's the cautionary footnote now, not a fourth option.

👮 The Policy: the container isn't the boundary that matters

The framework, and where it comes from

"Boundary, Policy, Lifecycle" as three separate questions for AI-agent sandboxing isn't an industry standard - it's from a single blog post by Luis Cardoso, a malware-analysis researcher, published in January. Worth saying plainly, because backlog notes and secondhand summaries have a way of making a good personal framework sound like consensus. It isn't, yet. But it's a genuinely useful way to cut the problem, and this year handed it a lot of supporting evidence.

What it says

Three separate questions, and most sandbox security effort only answers the first. Boundary: can the code physically escape the container, VM, or process? Policy: even fully inside the boundary, what is the agent actually allowed to touch - credentials, network egress, the host filesystem through a mount? Lifecycle: what happens to that access after the task that needed it ends? Most engineering attention goes to boundary. Most real incidents turn out to be policy.

The week that proved it

In July, security researchers documented a string of sandbox escapes across Cursor, Codex CLI, Gemini CLI, Antigravity, and Claude Code - written up as "the week of sandbox escapes." None of them were a broken container boundary. Every one had the same shape: an agent, still fully inside its sandbox, writes a file the boundary was never designed to police - a .vscode task config, a .claude hook, a modified Python virtualenv, git metadata under a nonstandard directory name that slips past path-based rules. Something outside the sandbox later reads and executes that file, unsandboxed. CVE-2026-35603 and CVE-2026-48124 came out of that wave. Anthropic fixed its case quickly; the write-ups describe the other vendors moving slower.

Why the framing matters

The container held in every one of these. Nobody broke out. The failure sat entirely in the second question - policy - which most teams never separately audit, because "we sandboxed it" already feels like the finished sentence.

What to actually check

Not "can the agent escape." Ask what it can still write to, reach over the network, or leave behind after the task ends, even while fully contained. That's three separate audits, not one.

Links

Four different kinds of "that wasn't actually solid." A dashboard that just stopped existing. A standard that only settles the envelope, not the substance. A free tool that's also a sales funnel, which is fine as long as you know it going in. And a sandbox that held perfectly while the policy around it didn't.

The last one is worth sitting with longer than a headline allows. Five different AI coding tools hit the identical failure shape in the same month, and the container was never the part that broke.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The Flux Kustomization that adopted a namespace it never should have touched

Ilia Gusev — Fri, 04 Sep 2026 14:01:17 GMT

Nobody touched the production Redis namespace that week. That was the whole problem with the postmortem: every line of the incident had a completely different name in it, valkey-test, and it still ended with resources gone from redis-system. Four unrelated decisions, each one reasonable on its own, lined up into a chain that let a Kustomization adopt a namespace it had never been told to manage, then delete it again on schedule, exactly as configured.

Nothing here is a Flux bug. Every step is the tool doing precisely what its config says. That's what makes it worth walking through slowly: four defaults compounding, not a single obvious mistake.

The four ingredients

The production Redis Kustomization watched path: ./kubernetes/, not ./kubernetes/production/redis-system/ - the whole tree above it, and not the directory that actually belonged to it. Flux Kustomization doesn't ask which manifests under that path belong to it, either. It takes all of them recursively, by default. Then there's targetNamespace, which doesn't filter what Flux picks up - it force-rewrites metadata.namespace on every namespaced resource it finds, regardless of what namespace the manifest author actually wrote. And prune: true, turned on for exactly the reason you'd turn it on: keep the cluster from accumulating dead resources nobody remembers creating.

Individually these are all sane defaults. path has to point somewhere. Recursion is what makes a directory of manifests useful instead of requiring one Kustomization per file. targetNamespace exists so you can reuse the same manifests across environments. prune is the thing that makes GitOps actually mean "the cluster matches Git," instead of "the cluster is a superset of Git that only grows."

How the adoption actually happened

Someone added a kubernetes/test/ directory for a Valkey experiment, unrelated to the Redis rollout, just organized under the same parent tree. The Redis Kustomization's next reconciliation walked ./kubernetes/, found the new directory, and pulled every manifest in it into its own managed set, including a namespace manifest for valkey-test.

targetNamespace: redis-system then did exactly what it's documented to do. For cluster-scoped resources like the Namespace object itself, the field is a no-op: Flux just created valkey-test as specified, a stray but harmless namespace. For anything namespaced sitting in that same test directory (a Deployment, a Secret), targetNamespace would have silently rewritten its namespace from valkey-test to redis-system, moving a test workload's manifests into the production namespace without anyone editing a single YAML file to say so.

Weeks later, someone cleaned up by renaming kubernetes/test/ to kubernetes-test/, moving it outside the watched path, which reads as the responsible thing to do. On the next reconciliation, Flux compared its desired state (Git, where the adopted resources no longer appeared under the watched path) against actual state (the cluster, where they still existed, because Flux had created them). prune: true did what it's for: deleted everything it had adopted and could no longer see in Git. Namespace/valkey-test, gone. Anything else it had silently claimed, gone with it.

The version where this isn't recoverable

What actually happened was close to the safe end of this failure mode: Flux deleted only what it had wrongly created itself, and the incident resolved as a strange namespace appearing and disappearing.

The version that doesn't resolve cleanly needs one more coincidence: a filename collision. If the wandering test directory had contained a secret.yaml that happened to share a name with a real production secret already managed under redis-system, one of two things happens instead. Either Flux's apply overwrites the real secret's contents with the test manifest's, silently, on the next reconcile, with no alert. Or, worse, the later prune reads the production secret as something it adopted and no longer sees in Git, and deletes the real one.

prune: true doesn't know the difference between a resource it should own and a resource it accidentally started owning. It only knows what's in Git and what's in the cluster under its watch. If the path is wide enough to accidentally include something it shouldn't, prune becomes, in the plainest sense, a weapon pointed at your own cluster by a rule that's doing exactly what it was told.

What actually closes the gap

Never point a Kustomization's path at a parent directory that might one day contain something unrelated. ./kubernetes/ is the mistake, ./kubernetes/production/redis-system/ is the fix: the deepest, most specific directory that contains only that Kustomization's own manifests and nothing else, ever.
If a shared parent path is genuinely unavoidable, a .fluxignore file at that path (same syntax as .gitignore) excludes subtrees like test/, staging/, or *-test/ from being swept up at all.
targetNamespace is a global override with no per-resource awareness, so stop reaching for it by default. A kustomization.yaml with an explicit namespace: field in the directory itself gives you the same result with none of the blast radius, because it only ever touches manifests that are already scoped to that directory.
spec.wait: true makes Flux wait for resources to report ready before considering the reconcile complete, which surfaces adoption-shaped surprises sooner, before they've had a chance to sit quietly for weeks.
A single YAML change won't close this gap on its own. Treat "do any two Kustomizations' paths overlap" as a question worth asking on a recurring basis, the same way you'd audit IAM policies for overlap: the failure mode here is two good configs that were never checked against each other, not a bad config in isolation.

Links

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

The 25% of your node that's gone before the first pod lands

Ilia Gusev — Wed, 02 Sep 2026 14:03:06 GMT

Somebody on my team once sized a node pool off the instance label. c5.4xlarge, 16 vCPU, done, that's what goes in the capacity plan. Then we actually looked at what kubectl describe node reported as allocatable, and it wasn't 16. It was closer to 15.5 on paper and noticeably less than that once the DaemonSets landed. Nobody had lied to us. We'd just never asked the node what it thought "capacity" meant.

That gap, between the number on the instance and the number a pod can actually schedule into, is usually 20-30% on a real cluster, and almost none of it shows up on a kubectl top dashboard. It's not a bug. Every layer taking a slice has a legitimate reason to exist. It's just that nobody sums the slices before doing capacity planning, so the plan is wrong by a fifth to a third before the first workload lands.

Here's where it actually goes.

kube-reserved and system-reserved

Every managed Kubernetes offering carves out CPU and memory for the kubelet, the container runtime, and the OS itself before your pods ever see it. The three big ones do it with three different formulas, and none of them show up on the labels on the instance type.

AKS is the most transparent about the math: kube-reserved is the lesser of 20 MB × max-pods + 50 MB or 25% of total memory. An 8 GB node running 30 pods reserves 650 MB before anything else happens. GKE publishes reference numbers too: a 2 vCPU / 7.5 GB node reserves roughly 70m CPU and 1,736 Mi memory as kube-reserved on its own. EKS doesn't publish an exact formula at all; the common community convention baked into most bootstrap scripts is memory=0.3Gi, ephemeral-storage=1Gi for system-reserved, which is a convention people copied into their tooling, not a number AWS commits to.

Three providers, three different answers to "how much of this node is actually mine," and the honest answer is you have to ask the node, not the instance catalog.

Eviction thresholds, and the version cliff nobody warns you about

Reserved capacity is the visible tax. Eviction thresholds are the one nobody budgets for: they carve out headroom that never shows up as "used" on any graph, because it's specifically the memory the kubelet refuses to let you use, held in reserve for the moment things go sideways.

GKE's default memory.available hard eviction threshold is 100Mi. AKS matches that on 1.29 and later. AKS before 1.29 defaulted to 750Mi, seven and a half times more headroom carved out of every single node, silently, as part of a version bump most teams treat as routine. If you fleet-upgraded across that boundary and your capacity math didn't move, it was wrong the whole time before and it's differently wrong now. EKS's commonly configured value in community bootstrap tooling sits around 200Mi, in between the other two, and again: not a published AWS default, a convention.

None of these thresholds are wrong to have. A node with zero eviction headroom OOM-kills workloads instead of gracefully shedding the least important pod first. The reservation itself is fine. What trips people up is that "allocatable" on the node object already accounts for kube-reserved, while the eviction threshold sits on top of that, invisible to anything reading the allocatable field. Most capacity math stops at allocatable and never goes further.

Pause containers: small, until you're at scale

This one is genuinely tiny per pod: a pause container holding the shared network and IPC namespace for everything else in the pod runs around 1 MB of resident memory. Nobody should reorganize a sprint around it.

But it's 1 MB times every pod on every node, all the time, for the lifetime of the cluster, and it's invisible in the same way the eviction threshold is: real memory the OS has committed, attributed to a container nobody thinks about when they're reading a namespace's usage graph. At 1,000 pods that's roughly a gigabyte of RAM doing nothing but holding namespaces open, spread across your fleet in amounts too small to alert on and too persistent to ever go away.

No budget breaks on it alone. What it does is quietly turn "sum of container requests" into a slightly wrong answer to "how much memory does this cluster actually need" - permanently, by a margin that grows with pod count.

DaemonSets: the capacity your FinOps dashboard calls "infrastructure"

This is the one that actually moves budgets, and it's the one most cost dashboards get structurally wrong.

CNI, CSI, the log shipper, the node-level monitoring agent, whatever security or service-mesh sidecar-injector runs as a DaemonSet - every one of them takes a CPU and memory reservation on every node in the cluster, multiplied by node count, for the entire life of the fleet. This cost scales with your node count, not your workloads, which means it moves in lockstep with your other costs, whatever's already driving the bill.

The reservation itself isn't what costs money. Most FinOps tooling attributes CPU and memory by namespace or by workload label, and DaemonSets frequently live in a kube-system or platform namespace that gets bucketed as "infrastructure" and never allocated back to the teams whose CNI policies, log volume, or security scanning drove the agent's resource needs in the first place - that's where the money actually goes missing. Five or six DaemonSets on a hundred-node cluster is a real, attributable cost that shows up on nobody's per-team invoice.

If your FinOps dashboard has a line called "infrastructure" that's bigger than any single team's line, that's usually where it's hiding.

Hosted control plane fees: the one that's actually on your bill

Everything above is invisible capacity tax. This one, at least, is a real line item - it's just inconsistently visible depending on who's cutting your check.

As of this year, EKS and GKE both charge $0.10 per cluster per hour for the control plane while your Kubernetes version is in standard support - about $73/month, though GKE offsets that with a $74.40 monthly credit per billing account that covers exactly one zonal or Autopilot cluster. AKS's free tier charges nothing for the control plane at all; the $0.10/hour tier buys you a 99.95% uptime SLA you don't get for free. And EKS has a second number worth knowing before you need it: once a cluster's Kubernetes version rolls into extended support, that $0.10/hour becomes $0.60/hour - six times the base rate, for the exact same control plane, as the penalty for not upgrading.

On bare metal, none of this exists as a line item, because the control plane is your hardware and your amortization schedule, which is precisely why it's easy to under-budget for it when you migrate the other direction.

What to actually do with this

kubectl top was never lying to you about usage. It was answering a question you weren't asking - "how much is running" instead of "how much of this node's sticker capacity did I actually get to use." Those are different numbers, and the gap between them is kube-reserved plus the eviction threshold plus pause-container overhead plus your DaemonSet count, none of which show up in the same place.

The concrete version: read kubectl describe node and compare Capacity to Allocatable yourself, on your actual node type, on your actual provider - don't trust the instance catalog. Then subtract your DaemonSet requests times node count before you tell finance what a node pool costs per unit of real workload capacity. That number, not the instance label, is the one your capacity plan should be built on.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

Issue #033 - TOON, zot, and the RCE Kubernetes calls working as intended

Ilia Gusev — Tue, 01 Sep 2026 14:03:31 GMT

Five deep-dive weeks in a row, and I promised the next open week would break the pattern. What landed in it doesn't share a vendor or a language, but it shares a shape: each of these four is a boundary someone drew somewhere you wouldn't expect. How much structure an LLM really needs. How much a registry needs to run at all. Where Prometheus Operator's declarative reach quietly stops. And what Kubernetes itself is willing to call a bug.

Grab bag's back.

🚀 Sandbox Watch: TOON

What it is

A serialization format built specifically for LLM context, not for storage or wire transfer. Token-Oriented Object Notation - TOON for short - shipped in October 2025 from a single maintainer, Johann Schopplich, MIT licensed. It hit Hacker News and picked up over 2,000 stars in three days. It's sitting past 25,000 now.

How it works

Not a compression trick. TOON borrows YAML-style indentation for nested objects, then switches to a CSV-like table layout the moment it sees an array of uniform objects - which is most of what actually gets shoved into a prompt: query results, API responses, tool-call output. Each array gets an explicit [N] row count and a {fields} header up front. That's not for humans. It's a guardrail the model uses to check its own parsing as it goes, instead of guessing where one object ends and the next begins from indentation alone.

The numbers, and where they stop working

The project's own benchmark: 46.3% fewer tokens than formatted JSON, at 70.1% accuracy on retrieval tasks. An independent run across four models - Claude Haiku, Gemini Flash, GPT-5 Nano, Grok - found 39.6% fewer tokens, with accuracy that came out higher than JSON's, 73.9% against 69.7%.

The authors are upfront about where it loses, too. Deeply nested or non-uniform data - minified JSON often wins outright. Semi-uniform arrays, somewhere around 40-60% field consistency, and the savings mostly evaporate. Purely tabular data - plain CSV is still smaller, TOON's explicit lengths and field lists cost you 5-10% over that. And on local or quantized models, some setups parse JSON faster in wall-clock time even while burning more tokens to do it.

When to reach for it

Not as a database format, not as an API contract - the authors call it a translation layer for LLM input, and that's the honest framing. If you're stuffing a hundred rows of uniform records into a context window before a completion call, it's worth trying. If your payload is one deeply nested object, don't bother.

💎 The Hidden Gem: zot

What it is

A vendor-neutral, OCI-native container registry. CNCF Sandbox since December 2022, still shipping - v2.1.19 landed in early August. About 2,700 stars, small next to Harbor's, but the commit graph is alive.

Where it differs from Harbor

I went looking for "zot has full OCI Distribution Spec support, Harbor doesn't" as the pitch, and that angle is stale - Harbor implements spec 1.1 too now, so that's not a real differentiator anymore.

The real one: zot runs with no database at all. Harbor needs Postgres behind it. For an edge deployment or anything resource-constrained, that one line is the whole pitch: one static binary, no stateful dependency to babysit. zot also has native support for the OCI Referrers API, which is the mechanism Cosign actually uses to store and look up signatures, so signature and SBOM attachment works as a first-class citizen rather than a bolted-on scan step.

The honest trade

Harbor still has a web UI. RBAC, project-based multi-tenancy, built-in vulnerability scanning, replication between registries, audit logging - and it's CNCF Graduated, zot is Sandbox. None of that is small. zot isn't trying to be a Harbor replacement; it's what you reach for when you want a registry to just serve OCI artifacts and nothing else runs on the same box.

CoreWeave uses zot as the registry behind their CKS Kubernetes service - it's in their own changelog, not a case study somebody paid for.

⚔️ The Showdown: the Probe CRD gap nobody's closed

The comparison I went looking for, and didn't find

I went in expecting a head-to-head: some operator wrapping Prometheus Blackbox Exporter versus using Blackbox Exporter directly. What I found instead is that the project going by that description - endpoint-monitoring-operator, 23 stars - isn't a Blackbox Exporter wrapper at all. It's its own thing: a separate EndpointMonitor CRD, its own probe logic for HTTP, HTTP-with-JSON-body-validation, TCP, DNS, ICMP, even Trino and OpenSearch health checks, and it alerts straight to Slack or email. It doesn't talk to Prometheus. So the comparison as pitched doesn't exist - which turned out to be more interesting than the comparison itself.

Where the real gap is

Prometheus Operator ships a Probe CRD for exactly this job - declaratively wiring blackbox-style checks to Blackbox Exporter, either a static target list or auto-discovered from your Ingress objects. It's mature, it's part of kube-prometheus-stack, most clusters running the standard observability stack already have it.

But it was built for simple, ping-style checks. Go looking at the project's own GitHub discussions and you'll find someone trying to describe hundreds of HTTP targets, each with its own headers and body, and hitting a wall - the Probe CRD has no way to express per-target Blackbox module configuration at that level. A maintainer's own answer in that thread: what's actually missing is "a CRD to deploy and manage Blackbox Exporter" itself, not just targets pointed at one. That issue has been open for years.

My read

That's why people write bespoke operators like endpoint-monitoring-operator instead of extending the Prometheus ecosystem: not out of preference, but because managing Blackbox Exporter's own module config was never made declarative. Anyone whose checks need real assertions - not "did we get a 200," but "does this JSON field say what it should" - ends up walking away from Blackbox Exporter entirely and building something narrower and custom. Nobody built a rival tool here. They just never wrote the CRD that would have made Blackbox Exporter's own config declarative.

👮 The Policy: the RCE Kubernetes calls working as intended

What it takes

This attack starts from one RBAC permission most clusters hand out without a second thought: get on nodes/proxy. No prior foothold on the node, no shell on the cluster required - the permission alone gets you in.

Security researcher Graham Helton published the mechanism in late January. The apiserver's nodes/proxy endpoint is meant to let you read things like node metrics through the apiserver instead of reaching the kubelet directly. The kubelet, on the other end, decides whether to allow a WebSocket upgrade based on the HTTP method of the handshake - and per RFC 6455, that method is always GET. Which means the secondary check that's supposed to gate an actual /exec - the create verb - never fires on the WebSocket path. get on nodes/proxy is enough, by itself, to open a connection to kubelet:10250 and run a command inside any pod on that node.

The 69 charts

Helton didn't stop at the theory - he audited what ships with this permission. Sixty-nine Helm charts grant nodes/proxy get as a matter of course, the full list with links is in his write-up. The names on it are the ones nobody questions: Prometheus, Grafana's Promtail, Datadog, Elastic Agent, Cilium, the OpenTelemetry Collector, Trivy Operator when its cluster role is enabled. Exactly the observability stack most platform teams install without reading the RBAC it asks for.

Working as intended

There's no CVE here, and there isn't going to be one. Kubernetes' Security Response Committee closed it on January 23rd, classified as Won't Fix, working as intended. In their own words, checking a verb at one layer and an HTTP method at another is "brittle, architecturally incorrect" - so instead of patching the specific hole, they're pointing at KEP-2862, Fine-Grained Kubelet API Authorization. It splits the kubelet API into separate, narrower permissions - nodes/metrics, nodes/stats, nodes/log, nodes/healthz, nodes/pods - instead of one nodes/proxy that covers all of it. It's Beta now, behind a feature gate, GA is targeted for April.

Worth being precise about how this differs from the kubelet-API piece I ran back in issue-015: that one assumed an attacker already sitting on the node network, reaching port 10250 directly. This one needs nothing but a ServiceAccount token carrying a permission your monitoring stack probably already has. No node access required at all - a different privilege boundary, not a rerun of the same finding.

What to do about it

Audit which service accounts in your cluster hold nodes/proxy. If you're running any of the standard observability charts, the answer is probably "more than you'd guess." Then treat KEP-2862's rollout as your real timeline, not a patch that's coming - because it isn't.

Links

Four boundaries, four different people drawing them. What TOON decided an LLM doesn't need. What zot decided a registry doesn't need. The CRD nobody at Prometheus Operator got around to writing. And the line Kubernetes drew around what even counts as a bug.

The last one's the one I keep coming back to - not because it got patched, but because it didn't, and it's probably sitting in your cluster right now if you're running anything that watches your nodes.

Questions? Feedback? Reply to this email. I actually read them.

- Ilia

LowCardinality: the dictionary stops at 8192

Ilia Gusev — Fri, 28 Aug 2026 14:04:54 GMT

Wrap a string column in LowCardinality and ClickHouse stops storing the string. It builds a dictionary of distinct values and stores small integer positions into it, so a GROUP BY compares integers instead of strings and the column compresses like integers too. On a country code or an HTTP method the win is large and free.

The advice attached to it is always some version of "use it under ten thousand distinct values". That number is real, and it's a rounded-off consequence of a setting most people have never looked at.

low_cardinality_max_dictionary_size = 8192

What the setting does

8192 is the maximum size, in rows, of the dictionary ClickHouse will write for a column. The documentation's blunt about what happens when you exceed it:

All the data that can't be encoded due to maximum dictionary size limitation ClickHouse writes in an ordinary method.

No error. No warning. No log line at the default level. Values that fit go in the dictionary as integer keys; everything past the limit is written as plain strings, in the same column, and the column keeps working exactly as before.

There's a companion setting that shapes what "past the limit" means:

low_cardinality_use_single_dictionary_for_part = false

With the default, when a dictionary fills up the server starts a new one rather than giving up. So a part with many distinct values ends up carrying several dictionaries plus whatever spilled over. Set it to 1 and you get exactly one dictionary per part and everything beyond 8192 stored the ordinary way.

Either way the outcome is the same in the direction that matters: the type stops being an optimisation and starts being an overhead, and nothing tells you.

Why the failure is quiet in both directions

The reason this catches people is that the type's behaviour degrades along two axes that don't move together.

Reads degrade gently. A partly-encoded column still answers queries; you just get less of the compression and less of the integer-comparison speedup than you expected, proportional to how much spilled.

Writes degrade immediately. Every insert has to hash the incoming value and look it up in the current dictionary, and that cost is paid whether or not the value ends up encoded. On a column with millions of distinct values you're paying full price for hashing on every row and getting nothing back.

Memory sits in the middle. Dictionaries live in RAM while parts are being read and merged, so a high-cardinality LowCardinality column adds resident memory in exchange for the storage saving it isn't delivering.

The documentation puts the far edge at a hundred thousand distinct values, past which the type "can perform worse in comparison with using ordinary data types". Between 8192 and 100k you're in a zone where it's neither clearly good nor clearly bad, and that zone is where nobody investigates.

Finding out which columns are lying to you

Cardinality per column is one query, and it's worth running against a real table rather than a sample:

SELECT
    uniqExact(status)      AS status_uniq,
    uniqExact(user_agent)  AS ua_uniq,
    uniqExact(request_id)  AS rid_uniq
FROM logs
WHERE event_date >= today() - 7;

Anything above 8192 that's currently declared LowCardinality is spilling. Anything comfortably under it that's a plain String is leaving a win on the table.

The subtlety is that cardinality is measured per part, not per table, and parts are built from inserts and merges. A column with 50,000 distinct values spread evenly across time might sit under 8192 within any single part and encode perfectly well, while the same 50,000 values arriving in one bulk load will not. Sorting order matters here too, since data ordered by the column groups its values into fewer parts.

Which means the honest test is measurement rather than arithmetic:

SELECT
    name,
    formatReadableSize(sum(data_compressed_bytes))   AS compressed,
    formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed,
    round(sum(data_uncompressed_bytes) / sum(data_compressed_bytes), 1) AS ratio
FROM system.columns
WHERE table = 'logs' AND database = currentDatabase()
GROUP BY name
ORDER BY sum(data_compressed_bytes) DESC;

A LowCardinality column whose compression ratio looks like its neighbouring plain strings isn't being encoded.

Enum is not the alternative people think it is

Enum8 and Enum16 do the same job with the value list fixed in the DDL, which makes them faster on insert - the mapping is static, so there's no hashing and no dictionary to grow - and completely rigid. A value that isn't in the enum is an error, and adding one means an ALTER TABLE.

That rigidity is a feature for a closed set: days of the week, a payment state machine you control, HTTP methods. It's a production incident for anything a third party can extend, because the day a new value appears your inserts start failing.

Enum16 also tops out at 65,535 values, which is comfortably above the point where LowCardinality has stopped helping. If you're choosing between them on capacity you've already picked the wrong tool for that column.

The limits worth knowing

LowCardinality isn't free at query time in every direction. Most string functions work through it transparently, but anything that has to materialise the actual strings - tokenisation, some regex paths, certain joins - decodes the whole column first, and for those the type costs you rather than saving you.

The setting is per-insert rather than per-column, which surprises people who go looking for a way to raise the limit on one wide column. Raising low_cardinality_max_dictionary_size globally trades RAM across every LowCardinality column in the server for the benefit of one, so the usual answer is to change the column type rather than the setting.

And the guidance shifts with version: this is ClickHouse 26.7 behaviour, and defaults in this area have moved before. Read the value out of your own server rather than trusting a number in a post:

SELECT name, value FROM system.settings WHERE name LIKE 'low_cardinality%';

Knative and PDB: protects nothing, blocks drains forever

Ilia Gusev — Wed, 26 Aug 2026 14:01:31 GMT

The advice sounds reasonable and it's everywhere: don't put a PodDisruptionBudget on a Knative service, because minAvailable: 1 will block scale-to-zero and you'll lose the thing you adopted Knative for.

It will not block scale-to-zero. It cannot. And that is worse, because people set it expecting protection and get a pod that disappears anyway plus a node they can no longer drain.

Why the autoscaler doesn't care

A PodDisruptionBudget is enforced in exactly one place: the Eviction API. When something calls POST /pods/{name}/eviction, the API server consults every PDB whose selector matches that pod and refuses if the budget would be violated. That is the entire mechanism.

Scaling down is not an eviction. When the Knative Pod Autoscaler decides a revision has no traffic, it writes to the Deployment's scale subresource - replicas: 0 - and the ReplicaSet controller deletes the pods directly. No eviction call, no PDB lookup, no budget check.

So the sequence people expect - PDB says one pod must stay, therefore Knative can't go to zero - never happens. The pod goes away on schedule and the PDB is not consulted at any point.

You can watch this from two terminals if you want to be sure. Send traffic, watch a pod appear, stop traffic, and watch kubectl get pdb report ALLOWED DISRUPTIONS 0 the whole time the pod is being removed underneath it.

What it does block

Node drains, which is precisely what a PDB is for, and precisely the interaction nobody plans for.

Take a service pinned with autoscaling.knative.dev/minScale: "1" because it can't tolerate cold starts, and a PDB with minAvailable: 1 because that's what you always write. There is one pod. The budget allows zero disruptions. kubectl drain on that node calls the Eviction API, gets a 429, retries, and keeps retrying.

Forever. The autoscaler has no reason to add a second pod, because there's no traffic pressure to justify one, and the drain has no way to ask for one. A cluster upgrade rolling through the fleet stops on that node and waits for a human.

That failure has a shape worth recognising: a PDB that permits zero disruptions on a workload that will never scale up on its own is a permanent drain block, and it is silent until the day somebody drains that specific node.

Then there's the revision problem

Even where a PDB is the right tool, Knative's object model fights the way you'd normally write one.

Revisions are immutable. Every configuration change makes a new Revision, each with its own Deployment and its own pod labels:

serving.knative.dev/revision: my-app-00042
serving.knative.dev/service:  my-app

Write a PDB against the revision label and it's dead weight the moment you deploy: it now selects a Revision that's scaling to zero while the new one runs unprotected. Keeping up means creating a PDB per deploy and garbage-collecting the old ones, which is a controller nobody wants to own.

Selecting on serving.knative.dev/service covers every revision of a service at once, including both sides of a traffic split, which is usually what you meant. It also means your budget arithmetic has to account for pods across two revisions during a rollout.

Where Knative's availability comes from

For a service that scales to zero, asking "how do I keep a pod available" is the wrong question, because the answer is that there isn't one and that's the design.

Requests to a zero-scaled revision don't fail. They go to the Activator, which buffers the request, tells the autoscaler to scale up, and forwards it once a pod is ready. Your availability during that window is the Activator's availability, not your pod's.

Which is why the PDBs that matter for a Knative install are the ones on Knative's own components, and they already exist. A default install runs the activator, autoscaler, controller and webhook with two replicas, and ships an activator-pdb with minAvailable: 70%. If you're going to spend attention on disruption budgets in a Knative cluster, spend it verifying those survived your install tooling rather than adding new ones to application revisions.

What to do instead

For most services: nothing. Scale-to-zero and the Activator are the availability story, and a PDB adds a drain hazard in exchange for no protection.

For a service that must always be warm, set minScale: 2 rather than 1. Two pods across two nodes give a drain somewhere to move work to, and only then does a budget make sense - written as maxUnavailable: 1 rather than minAvailable: 1, because that shape always permits one disruption regardless of how the replica count moves under you.

If you inherited a cluster and want to know whether this is already waiting for you:

kubectl get pdb -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed \
  | awk '$3=="0"'

Anything listing zero allowed disruptions is a node drain that will hang. Cross-check those against Knative services with minScale: 1.

The limits worth knowing

PDBs only ever governed voluntary disruptions. A node that dies takes your pod with it whatever the budget says, and no arrangement of PDBs changes that - the guarantee was always about planned maintenance.

Knative also isn't unique here. Any autoscaler that writes the replica count directly, HPA included, moves pods without consulting a PDB. The Knative case stands out because scale-to-zero makes the gap between "budget says one" and "reality says none" a routine daily event rather than an edge case.

And this is Knative Serving 1.23 behaviour. Component defaults, including the activator PDB, are the kind of thing that shifts between releases, so check yours rather than trusting a number in a post.

Issue #032 - Keyless signing: nobody manages signing keys anymore

Ilia Gusev — Tue, 25 Aug 2026 14:02:01 GMT

For about twenty years, code signing meant key custody. Generate a key, protect it for its lifetime, and accept that losing it costs you your identity while leaking it costs everyone else theirs. Every practice around it - HSMs, air-gapped signing machines, elaborate ceremonies - existed because the key was a long-lived secret that had to stay secret.

That problem is gone. Not mitigated, gone: in a Sigstore signature there's no key to keep, because the key is generated, used once, and thrown away inside the same command.

Which is a genuine win, and it moved the failure somewhere less visible. I covered Sigstore as one block in issue #005 back in February, at the level of "cosign signs your images". This is the other half: what you're actually trusting once the key is gone, and why the interesting bugs now live in the verify path rather than the sign path.

🏗️ Architectural Pattern: a certificate that expires before you finish reading this

The exchange

The flow is short enough to hold in your head.

You authenticate to an OIDC provider - Google, GitHub, your own issuer - and get an ID token. Cosign generates a throwaway keypair locally. It sends the public key and the token to Fulcio, the CA. Fulcio validates the token against the provider, then issues an X.509 certificate binding that public key to the identity in the token: the email or workflow reference goes in the Subject Alternative Name, and the OIDC issuer goes in a dedicated extension.

The certificate's valid for ten minutes.

You sign, the private key is discarded, and the signature plus the certificate go into a Rekor entry, which timestamps the fact that this certificate signed this digest while the certificate was still valid.

Why ten minutes is the point

A ten-minute certificate can't be revoked in any meaningful sense, and it doesn't need to be. Revocation infrastructure - CRLs, OCSP, the whole apparatus - exists because certificates outlive the circumstances that justified them. Shrink the window below the time it takes to notice and respond to a compromise and the problem stops being worth solving.

The cost is that a ten-minute certificate is useless as a verification anchor on its own. By the time anyone checks your signature, the certificate expired months ago. So verification can't ask "is this certificate valid now". It asks "was this certificate valid when the signature was made", and the only reason that question has an answer is the transparency log entry that timestamped it.

That inverts a dependency people don't expect. In classic signing, the log is an audit nicety. Here it's load-bearing: without a log entry, an expired certificate proves nothing.

What you're trusting instead

The key is gone and the trust didn't evaporate, it moved. A verifier is now trusting, in order: that the OIDC provider correctly authenticated whoever asked; that Fulcio correctly validated the token; that the log entry is genuine and inclusion-proven; and that the identity in the SAN is one you meant to accept.

The last one is yours. The first three belong to somebody else, which is fine - that's what a public good looks like - but it's worth saying plainly, because "keyless" gets read as "trustless" often enough to matter.

🆚 The Showdown: Rekor v2 against the log it replaced

Rekor v2 reached GA this year, and it's an unusual upgrade in that most of the release notes are things being taken away.

Gone: signed timestamps returned by the log. Gone: the search index. Gone: entry types intoto, rekord, helm, tuf, rfc3161, jar, rpm, cose and alpine, leaving hashedrekord for artifacts and dsse for attestations. Gone: attestation storage, so you keep your attestations next to your artifacts like any other file.

Underneath, the Trillian backend was replaced with a tile-based log built on Tessera, and each shard now has its own URL along the lines of logYEAR-rev.rekor.sigstore.dev rather than hiding behind one endpoint.

The case for deleting things. A transparency log's only job is to make it impossible to sign something without leaving a permanent, publicly auditable trace. Every additional feature - a search index, ten entry formats, blob storage - is surface that has to stay correct forever, because a transparency log you have to migrate off is not much of a transparency log. Cutting to two entry types and a tile-based backend makes it cheap enough to run that other people can run their own, which is the actual goal.

The case against, and it's real. The search index going away means you can no longer ask the log "show me everything signed by this identity" without building that index yourself. For anyone doing detection - watching for signatures made by an identity that shouldn't be signing - that was the useful interface, and its replacement is "coming later as a separate verifiable service".

The client-side consequence is the one to plan for: shard URLs must come from the TUF root, not from your config. Hardcode a log URL today and you've built a thing that breaks on the next rotation. Rekor v1 keeps running in parallel and will eventually freeze for new uploads, with a year's notice.

👮 The Policy: the verify command is the whole security boundary

Signing is now easy enough that people do it. Verification is where the work moved, and it's the half that gets copy-pasted.

Cosign v2 made two flags mandatory for keyless verification, and the reason is worth internalising:

cosign verify \
  --certificate-identity 'https://github.com/acme/api/.github/workflows/release.yml@refs/heads/main' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/acme/api:1.4.2

Without those, "verify" means "some identity that Fulcio was willing to certify signed this", which is every GitHub account in existence. The command would succeed on an image signed by an attacker with a free account, and it'd look exactly like success.

The regex habit is the same bug wearing a hat. --certificate-identity-regexp='.*' shows up constantly, usually added to make a pipeline go green. It restores precisely the property the mandatory flags were introduced to remove. If you need a pattern, anchor it: pin the issuer exactly, and let the regex cover only what varies, like a tag or a branch under a repository you control.

Pin the workflow, not the repository. For CI identities the SAN contains the workflow file and the ref. Accepting any workflow in a repository accepts the one an attacker adds in a pull request, so the identity you pin should name the release workflow on the branch you protect.

The verification path is where the bugs are

If that sounds theoretical, January settled it. CVE-2026-22703 let a crafted bundle verify successfully even though its embedded Rekor entry didn't reference the artifact's digest, signature or public key. Cosign checked the log entry's signature and skipped the comparison that ties the entry to the thing in front of it, so any valid entry would do.

The practical effect is that an attacker who already had your identity could sign something and attach a Rekor entry pointing at an unrelated event, which removes the audit trail - the one property the log exists to provide.

Two details make it more instructive than an ordinary CVE. It affected cosign v2 users on default flags, while v3 users on default flags were unaffected. And it was a regression: the same class of bug had been fixed before and came back through a refactor.

Fixed in v2.6.2 and v3.0.4. Current releases are v3.1.3 and v2.6.5, and if your CI's pinned to a cosign version from last year, that's this week's fifteen-minute job.

The honest part

None of this makes an artifact good. A verified signature says a particular identity signed a particular digest at a particular time, and that's all it says. A compromised CI account produces perfectly valid signatures over malicious images.

What it buys you is attribution and an audit trail nobody can edit without being seen, which is a large improvement over an unsigned artifact and a smaller one than the word "verified" suggests.

And the monitoring gap is real. Sigstore's value proposition assumes somebody notices when an identity signs something it shouldn't. With the search index gone from Rekor v2, the tooling for that is thinner than it was, so if your threat model depends on detection rather than on after-the-fact forensics, that's a project rather than a flag.

Where the difficulty went

The thing I keep noticing about this transition is that it didn't remove difficulty, it relocated it into a place with worse feedback.

Key custody was hard, and it was hard in a way that announced itself. Everyone knew the HSM was a chore. Nobody accidentally believed they had good key management when they didn't.

Identity-based verification is easy to get wrong and comfortable to leave wrong, because a misconfigured verify command returns zero and prints a checkmark. The pipeline is green, the dashboard says signatures are enforced, and the policy admits anyone with a GitHub account. There's no chore to remind you.

We checked ours after reading the January advisory. Two of nine pipelines pinned issuer and identity properly. Five used a regexp with .* somewhere in it. Two verified with --insecure-ignore-tlog, which somebody added during an incident in March and nobody removed.

Questions? Feedback? Reply to this email. I actually read them.

Ilia

Topology-aware routing: 7 replicas work, 8 don't

Ilia Gusev — Fri, 21 Aug 2026 14:01:48 GMT

Cross-zone traffic is billed per gigabyte, so the annotation that keeps requests inside one availability zone looks like free money:

metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto

You apply it, the bill doesn't move, and there's nothing in kubectl describe service that looks like an error. The usual conclusion is that the feature is broken or that the cluster is missing zone labels.

Neither. The controller looked at your service, decided that routing by zone would overload something, and declined. It even said so, in an event nobody reads.

What the controller is actually deciding

The EndpointSlice controller doesn't route anything. It writes hints into EndpointSlices - a note on each endpoint saying "this one is for zone X" - and kube-proxy on each node honours the hints by restricting its endpoint set to the local zone.

It decides who gets a hint from allocatable CPU per zone, and pod counts never enter into it:

The controller allocates a proportional amount of endpoints to each zone. This proportion is based on the allocatable CPU cores for nodes running in that zone.

A zone with twice the CPU is expected to receive twice the traffic, so it should hold twice the endpoints. From that expectation the controller computes, per zone, the smallest endpoint count it would tolerate:

const overloadThreshold float64 = 0.2

desired := ratio * float64(numEndpoints)
minimum := int(math.Ceil(desired * (1 / (1 + overloadThreshold))))

Read that as: each endpoint may end up carrying at most 20% more than its fair share. Anything worse and the controller refuses. It sums minimum across zones, and if the total exceeds the number of endpoints you actually have, it gives up and leaves the service on cluster-wide routing.

overloadThreshold is a package-level constant and it isn't exported. There's no flag, no annotation value, no per-service override.

Adding a replica can turn it off

Because minimum is rounded up per zone, the outcome isn't monotonic in replica count. Adding a replica can turn the feature off.

Three zones with equal allocatable CPU, varying the number of ready endpoints:

 3  hints
 4  refused
 5  refused
 6  hints
 7  hints
 8  refused
 9  hints
10  hints
11  refused
12  hints

Eight is the one that catches teams, because eight is a normal number of replicas and seven works. Walk through it: each zone's fair share is 8/3 = 2.67 endpoints, the minimum is ceil(2.67 / 1.2) = 3, three zones need 9, and you have 8. Refused.

Nine replicas works, which is where the "at least 3 per zone" rule of thumb comes from. That rule is a decent summary of the safe region and a bad description of the boundary, and the boundary is where the surprise lives - an HPA that scales 7 to 8 silently disables zone routing, then re-enables it at 9.

Unequal CPU across zones shifts the whole pattern. On a cluster where one zone runs bigger nodes, the working counts differ from the table above, so treat it as a shape rather than a lookup.

Finding out which case you're in

The controller emits a Warning event on the Service with reason TopologyAwareHintsDisabled, and the message names the specific reason:

InsufficientNumberOfEndpoints - fewer endpoints than zones
MinAllocationExceedsOverloadThreshold - the arithmetic above didn't work out
NodesReadyInOneZoneOnly - only one zone has ready nodes
NoZoneSpecified - an endpoint's node has no zone label

So the first diagnostic step is not tcpdump:

kubectl get events --field-selector reason=TopologyAwareHintsDisabled -A

If nothing comes back and the hints still aren't there, check whether the hints exist but kube-proxy is ignoring them:

kubectl get endpointslice -l kubernetes.io/service-name=web \
  -o jsonpath='{range .items[*].endpoints[*]}{.zone}{" -> "}{.hints}{"\n"}{end}'

Empty hints with no event usually means the controller never considered the service at all - the annotation is misspelled, or it's on the Deployment instead of the Service.

The newer field is a different mechanism

spec.trafficDistribution shows up in every discussion of this, and reading it as a tidier spelling of the annotation is a mistake worth avoiding.

They're separate code paths in the same controller. The annotation goes through the capacity heuristic described above. trafficDistribution goes through its own reconciler, whose entire policy is one comment in the source:

update adds a same zone topology hint for all ready endpoints

Every ready endpoint gets a hint for its own zone. No CPU ratios, no overload threshold, no minimum endpoint count, no refusal path. PreferSameNode does the same thing at node granularity.

Which means the migration people describe as cosmetic changes the behaviour in both directions. You gain a feature that actually turns on at 8 replicas. You lose the guard that was protecting a thin zone from taking a full share of traffic with one pod in it.

There's a precedence rule too, and it's silent: the reconciler computes canUseTrafficDistribution as valid field and annotation absent. Leave the old annotation on a Service while adding the new field, and the field is ignored entirely. No event, no warning, no status. The source carries a note that the whole branch goes away once the annotation is deprecated under KEP-4444, so the annotation is on its way out - but while both exist, the old one wins.

The assumption underneath all of it

Both mechanisms allocate endpoints by where capacity is. Neither knows where traffic comes from.

That's fine when request origin roughly tracks capacity, which is the common case for internal service-to-service calls in a balanced cluster. It falls apart when a single zone originates most of the traffic - a batch job, a cron-driven fan-out, an ingress tier that isn't spread the way your workloads are. Then zone-local routing concentrates that load onto one zone's endpoints while the others idle, and the cost saving comes out of your tail latency.

This is also the second half of a two-hop story. externalTrafficPolicy governs the first hop, from the external load balancer to a node, and it has its own skew problem. Topology hints govern the second hop, from the node to a pod. They compose, and they fail independently.

The limits worth knowing

Hints are ignored entirely when internalTrafficPolicy: Local is set on the Service. The two features want the same decision made in different places, and the node-local policy wins.

The controller excludes control-plane nodes from its CPU ratios, ignores unready nodes, and pays no attention to taints or tolerations. A zone whose nodes are tainted so your workload can't schedule there still counts toward that zone's expected share, which is a good way to get refusals you can't explain from the pod distribution alone.

And if any single endpoint is missing a hint while others have them, kube-proxy falls back to all endpoints for that service. Partial hints aren't partially applied - it's all or nothing, per service, on every node.

Informer resync: the API call that never happens

Ilia Gusev — Wed, 19 Aug 2026 14:01:18 GMT

The controller was acting on stale data, so somebody lowered resyncPeriod from ten minutes to thirty seconds. It didn't help. Then to five seconds, which didn't help either, and made the process burn CPU on a schedule.

Resync doesn't fetch anything. From the client-go docs on AddEventHandlerWithResyncPeriod:

The resync operation consists of delivering to the handler an update notification for every object in the informer's local cache; it does not add any interactions with the authoritative storage.

Every word of that is load-bearing, and the last clause is the one that gets skipped.

What an informer is made of

Four pieces, and they're easier to reason about named:

A reflector does one LIST against the API server, then opens a WATCH from the resourceVersion that LIST returned. It's the only component that talks to the API server at all.

A DeltaFIFO queues what the reflector produces, as typed deltas: Added, Updated, Deleted, Sync.

An Indexer is the local cache - a thread-safe store with secondary indexes, so your controller can ask for "all pods on node X" without a query leaving the process.

Event handlers are your code, called as deltas drain out of the queue.

Everything a controller reads comes from the Indexer. client.Get() through a cached client, a lister, the informer's GetStore() - all of it is memory, populated by that one watch stream. That's the whole point of the pattern: one watch per resource type per process, shared across every controller in the binary, instead of N controllers polling.

What a resync actually does

The current client-go handles a full resync in five lines, and they answer the question completely:

case SyncAll:
    objs := clientState.List()
    for _, obj := range objs {
        handler.OnUpdate(obj, obj)
    }

clientState is the local store. OnUpdate(obj, obj) passes the same object as both the old and the new value.

So a resync walks your cache and calls your update handler once per object, with old and new identical. Nothing is fetched. Nothing is compared. If your handler starts with a check like "did the spec change since the last version" it will find that it didn't, every time, for every object.

Two consequences fall out of this immediately.

A handler that diffs old against new does nothing on resync. If you wrote your controller to skip work when nothing changed - which is a good habit - resync is a no-op for you, and turning it down will never make anything fresher.

A handler that enqueues unconditionally does full work on resync. Cache of fifty thousand pods, resync every thirty seconds, and you've signed up for fifty thousand reconcile enqueues every thirty seconds. The API server sees none of it. Your CPU graph sees all of it.

Then what is it for

Resync exists for drift between your cache and something the cache can't observe.

A controller that creates cloud load balancers can't see someone deleting one in the console. No Kubernetes object changed, so no watch event fires, and the controller has no reason to look. A periodic resync re-runs reconciliation for everything and gives it that reason.

That's the honest use case, and it's why the sensible values are minutes, not seconds. Ten to thirty minutes covers the drift case. Anything shorter is usually somebody trying to fix a freshness problem that resync has never addressed.

If you have no external state to drift against, zero is a legitimate setting. Plenty of controllers run with resync disabled and are correct, because the watch already delivers every change to every object they care about.

When the data really does go stale

Two failure modes, and neither is fixed by resync.

The watch broke and the reflector is re-listing. Watches die - connection resets, apiserver rollouts, a resourceVersion too old for the watch cache to serve. The reflector handles this by re-listing and starting a new watch, and during that window your cache is whatever it was. Watch HasSynced, which the source is explicit about:

HasSynced returns true if the shared informer's store has been informed by at least one full LIST of the authoritative state of the informer's object collection. This is unrelated to "resync".

You read your own write. You update an object through the API and immediately read it back from the cache. The write went to etcd; the cache updates when the watch event arrives, which is milliseconds later but not zero. Controllers that assume read-after-write against a cached client have a race, and it's the most common informer bug I've seen. If you need the object you just wrote, either read through an uncached client or structure the reconcile to be re-entrant and let the next event carry the new state.

The expensive part is the LIST

Resync is free for the API server. The initial LIST is not.

Historically the apiserver assembled the entire collection in memory before sending a byte of it, so a controller restarting against a hundred thousand pods produced a large allocation spike on the control plane, and a cluster full of controllers restarting together produced several at once.

That's what streaming lists fix. Under KEP-3157 the initial state arrives as a watch that streams items individually, giving the apiserver constant memory overhead instead of one proportional to the collection. Server-side it's the WatchList gate; client-side it's WatchListClient, which reached beta and is on by default. Both are beta rather than GA, so it's worth knowing which of your controllers actually use it before you take credit for the memory graph.

The apiserver side of this story - how watch requests get served without touching etcd at all - is the watch cache, and it's a separate mechanism from anything in this piece.

The limits worth knowing

Resync periods are per handler, but not freely. A shared informer has one resync check period, and a handler asking for a shorter interval than that gets the informer's, not its own. Requesting two seconds on an informer configured for ten minutes gets you ten minutes.

Shared informers are also shared in a way that bites: the informer factory dedupes by resource type, so two controllers asking for pod informers get the same one, along with each other's resync behaviour and each other's cache memory. That's usually what you want and it's occasionally why a memory number doesn't match anyone's mental model of who allocated it.

And the cache holds full objects. An informer on pods in a large cluster is holding every pod spec and status in your process. Transform functions on the informer let you strip fields you'll never read before they hit the store, which is the cheapest memory win available to most controllers and one almost nobody applies.

Issue #031 - Ambient mesh: the sidecar was a five-year detour

Ilia Gusev — Tue, 18 Aug 2026 14:02:54 GMT

Ambient mode went GA in Istio 1.24, in November 2024. That's twenty-one months ago. Istio is on 1.30.3 now, and somewhere in that stretch the question quietly flipped: it stopped being "should we try ambient" and became "why are we still injecting sidecars".

I covered ambient in issue #002 back in January, at the level of the sidecar tax - one Envoy per pod, fifty to a hundred megabytes each, startup ordering problems. That framing is fine and it's also the least interesting thing about the architecture. The resource number is a consequence. The mechanism underneath is stranger than "we moved the proxy to the node".

🏗️ Architectural Pattern: how a node proxy pretends to be a sidecar

The file descriptor trick

The obvious way to build a node-level proxy is to run it on the node, in the host network namespace, and route pod traffic to it. That design has a well-known problem: once the packet leaves the pod, you've lost the pod. Source addresses get rewritten, the proxy has to reconstruct which workload sent what, and every policy decision downstream depends on that reconstruction being right.

Ambient does something else. When a pod joins an ambient-enabled namespace, the istio-cni node agent enters the pod's network namespace, installs redirection rules there, and then hands ztunnel a file descriptor referring to that namespace over a Unix socket.

Ztunnel uses the descriptor to open listening sockets inside the pod's network namespace. One logical proxy instance per pod, sockets that live in the pod's own netns, a process that lives on the node.

So from the pod's point of view the proxy is local. From the node's point of view there's one Rust process instead of a hundred Envoys. The pod identity never has to be reconstructed, because the traffic never left the pod's namespace unattributed.

Three ports do the work, and they're worth knowing when you read a ss output at 2am:

15001   outbound, before HBONE encapsulation
15006   inbound plaintext
15008   inbound HBONE

Inbound TCP is redirected to 15008 or 15006 depending on whether it arrived encrypted. Outbound goes to 15001 and leaves the node wrapped in HBONE, which is HTTP CONNECT over mTLS - a tunnel that carries the original destination as connection metadata rather than as a rewritten address.

The identity problem, and the fix

In the sidecar model, the Envoy holding a workload's certificate ran in that workload's pod. The blast radius of a compromised proxy was one pod, and the certificate it held was its own.

Ztunnel holds a certificate for every unique service account among the pods on its node. It has to - it terminates mTLS on their behalf. A node running forty pods across twelve service accounts has one process holding twelve workload identities.

The guardrail is on the CA side. Ztunnel authenticates to the control plane as itself and then requests certificates for other identities, and the CA enforces that it may only do that for identities actually scheduled on that node. Requests for identities not running there are rejected.

That's a sound design and it is a genuinely different threat model from a sidecar. It's node-scoped rather than pod-scoped, and the thing standing between a compromised ztunnel and a cluster-wide identity grab is a control-plane check rather than the absence of the key material. If your security review asked "what changed", that's the answer, and it's better stated plainly than discovered later.

Where waypoints sit

Ztunnel does L4 and only L4. mTLS, L4 authorization, TCP-level telemetry. It doesn't parse HTTP headers, by design - that's how it stays small and how it stays Rust.

Anything at L7 needs a waypoint, which is a real Envoy, deployed as a normal Deployment for a namespace or a service account. The path becomes:

source pod -> source ztunnel -> waypoint -> destination ztunnel -> destination pod

Four hops where a sidecar mesh had two. That's the trade you're making for the resource savings, and whether it matters depends entirely on how much of your traffic actually needs L7 treatment.

🆚 The Showdown: what the sidecar was still doing for you

The migration guides are cheerful. The list below is the one I'd want before committing, because every item on it is a thing that worked yesterday and needs a decision today.

Per-workload EnvoyFilter and WASM. In sidecar mode you could attach a filter to one workload, because that workload had its own Envoy. Ambient has no per-pod Envoy to attach to. Filters either become global or move into a waypoint, and a waypoint serves a namespace or a service account rather than a single pod. Teams with one bespoke filter for one service feel this immediately.

L7 telemetry by default. A sidecar emitted HTTP metrics, logs and traces for everything passing through it, because everything passing through it was already parsed. Ztunnel gives you L4: bytes, connections, TCP-level authorization decisions. Your request-rate-by-response-code dashboard goes quiet for any workload without a waypoint. This is the single most common surprise, and it surprises people after the migration rather than during it.

Locality-aware routing. A sidecar carried locality information and preferred endpoints in its own zone without being asked. In ambient, traffic that goes through a waypoint goes to wherever that waypoint is, so a waypoint in the wrong zone turns a same-zone call into a cross-zone one - with the latency and the inter-AZ transfer bill that implies. The fix is deploying a waypoint per zone, which is fine, and which nobody plans for on the first pass.

VirtualService. Gateway API is the preferred routing surface in ambient. VirtualService support is alpha there. If your routing rules are years of accumulated VirtualService, that's a translation project, not a label change.

Multi-cluster. Stable in sidecar mode. Available in ambient with limitations that move release to release, which is the polite way of saying check the notes for your exact version rather than trusting a blog post.

And the resource claim deserves an honest reading. "Up to ninety percent" is real arithmetic under the right conditions: it's a ratio between one node proxy and N sidecars, so it scales with pod density and it shrinks the moment you deploy waypoints, because a waypoint is an Envoy again. A cluster with thirty pods a node and L7 policy everywhere saves much less than a cluster with a hundred pods a node running L4 mTLS only. Both are ambient. The savings differ by an order of magnitude.

The failure mode nobody mentions

A sidecar failing takes down one pod. Ztunnel is a DaemonSet, and a ztunnel restart interrupts every meshed pod on that node.

In practice this is mostly fine, because ztunnel restarts are fast and connections re-establish. It's still a different operational shape, and it belongs in the same conversation as the identity scoping: you traded per-pod failure domains for per-node ones, in exchange for not having to restart every application pod to upgrade the mesh.

👮 The Policy: migrating without a flag day

The good news is that the migration is genuinely incremental, and not in the marketing sense. Sidecar pods and ambient pods coexist in one mesh. That means the unit of migration is a namespace, and the rollback is removing a label.

A sequence that works:

Start by turning on L4 for one namespace and changing nothing else. Label it, and you get mTLS and L4 authorization with no pod restarts and no waypoint. This step costs almost nothing in resources and it's where most of the security benefit lives. If you stop here permanently for most namespaces, that's a legitimate end state rather than a half-migration.

Then work out which services need L7. Not which ones have an Envoy today - which ones use a feature that requires parsing HTTP. Header-based routing, JWT claims in authorization policy, retries and timeouts on HTTP semantics, request-level telemetry that someone alerts on. Usually a much shorter list than the number of workloads currently running sidecars.

Deploy waypoints for those and size them like anything else you own. A waypoint is a Deployment: it needs requests, limits, an HPA if the traffic warrants it, and a copy per zone if you care about locality. Treating it as invisible infrastructure is how you get a mesh-wide latency regression out of one under-provisioned pod.

Sort out the dashboards before the cutover, not after. Know which panels are fed by L7 metrics from sidecars, and decide for each one whether it moves to a waypoint or gets rebuilt on L4 signals. Leaving it until afterwards buys you a window where nobody can see anything, and that window is usually when somebody asks whether the migration broke something.

The honest part

This isn't free and it isn't instant. The resource win is real, the operational win of not restarting pods to upgrade the mesh is real, and both are paid for with an extra hop for L7 traffic, a node-scoped identity surface, and a translation project for anyone deep in VirtualService or per-workload filters.

There's also a version of this decision that ends with "neither". If you're running a mesh for mTLS and nothing else, and you're on a cloud CNI that can do transparent encryption between nodes, the honest comparison isn't sidecar against ambient - it's mesh against no mesh. Ambient makes the mesh cheap enough that the question is worth asking again, which is not the same as answering it yes.

What the detour was actually for

Calling the sidecar a five-year detour is unfair in one direction and accurate in the other.

It was unfair because the sidecar proved the model. Transparent mTLS, policy that doesn't live in application code, telemetry nobody had to instrument - all of that had to be demonstrated before anyone would accept a new component in the datapath at all, and the sidecar demonstrated it in the only way available at the time. The kernel primitives ambient depends on weren't there in 2018.

It's accurate because the cost model was wrong from the start, and everyone knew it. One proxy per pod is O(pods) for a job that's O(nodes) plus a bit of L7. We spent five years paying a linear tax for a constant-ish problem, and building tooling to make the tax bearable: injection webhooks, startup ordering, the whole holdApplicationUntilProxyStarts genre of workaround.

The pattern generalises past meshes, and it's the thing I'd take away if I only took one. When a piece of infrastructure needs a lot of tooling to make its cost model tolerable, the tooling is a signal about the cost model, not a solution to it. Sidecar injection webhooks were a very good answer to a question that shouldn't have been asked.

I went and counted our own sidecars after Istio 1.30 shipped. Two hundred and eleven Envoys, of which eleven are attached to workloads that use a single L7 feature. The other two hundred are doing mTLS.

Questions? Feedback? Reply to this email. I actually read them.

Ilia

externalTrafficPolicy Local: client IP for skewed load

Ilia Gusev — Fri, 14 Aug 2026 14:00:30 GMT

Every access log said 10.244.1.x. Rate limiting was useless, the geo rules did nothing, and the abuse dashboard showed one very busy internal address.

The fix is one line, and everyone finds it within an afternoon:

spec:
  externalTrafficPolicy: Local

Real client addresses come back immediately. What arrives later is a load distribution nobody asked for.

Why the addresses were wrong

Default policy is Cluster. Traffic hits any node, and kube-proxy forwards it to a pod wherever that pod lives.

That forward is the problem. If the packet arrives at node A and the pod is on node B, node B's reply has to travel back through node A, because node A is the one holding the connection with the load balancer. To make the return path work, node A rewrites the source address to its own. Your pod sees the node, not the client.

Local tells kube-proxy to stop forwarding. If there's a pod for this Service on the node the packet landed on, deliver it locally, unmodified. If there isn't, drop it. No cross-node hop means no return-path problem means no rewrite, and the client address survives.

The dropped-packet half sounds alarming and mostly isn't, because cloud load balancers health-check the NodePort. Nodes without a local pod fail the check and stop receiving traffic. The mechanism is doing exactly what it says.

The skew

Here's what the health check can't fix.

The load balancer distributes across the nodes that pass the check, and it distributes evenly, because it has no idea how many pods each one is running. It's balancing nodes. You care about pods.

Three nodes pass the health check. Node A runs one pod, node B runs one, node C runs four. The load balancer sends a third of the traffic to each node.

Node A's single pod gets 33% of all traffic. Each of node C's four pods gets about 8%.

A four-to-one spread across identical replicas, from a scheduler decision nobody made deliberately. It shows up as some pods pinned at high CPU while others idle, an HPA that scales on average utilisation and therefore scales at the wrong time, and p99 that tracks whichever pod drew the short straw.

Nothing is broken. Every component is doing its job. The mismatch is that one layer balances nodes and the layer you care about is pods.

Making the distribution match

Two shapes work, and both come down to making pods-per-node uniform.

A DaemonSet gives you exactly one pod per eligible node by construction, which makes node-balancing and pod-balancing the same thing. This is why ingress controllers are so often deployed as DaemonSets, and it's the cleanest answer if the workload is a proxy tier you want on every node anyway.

Otherwise, topologySpreadConstraints with maxSkew: 1 over kubernetes.io/hostname:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: web

DoNotSchedule rather than ScheduleAnyway matters here. The soft version lets the scheduler pile pods onto one node when things get tight, which is precisely the state you're trying to prevent.

Both of these have a cost the docs are quiet about: you've coupled your replica count to your node count. Scaling to 7 replicas across 3 nodes gives you 3/2/2 and a 50% spread that no constraint can remove.

The feature that is not the answer

Kubernetes has grown a newer field that comes up in every discussion of this, and it solves a different problem.

spec.trafficDistribution takes PreferSameZone or, since 1.35, PreferSameNode. The older PreferClose is still accepted and is now explicitly deprecated in the source in favour of PreferSameZone, which means the same thing with a name that says what it does.

It's tempting to read PreferSameNode as a gentler Local. It isn't, for two reasons.

It doesn't preserve the client address. trafficDistribution influences which endpoint kube-proxy picks. It has nothing to say about SNAT, and for NodePort and LoadBalancer traffic the client address still gets rewritten. If source IP is what you came for, this field does not deliver it.

And it's a hint. The API comment is unambiguous: implementations "can use this field as a hint, but are not required to guarantee strict adherence." Local is a rule with a defined failure mode, which is what you want from something load-bearing.

What trafficDistribution does help with is cross-zone data transfer cost and the latency of an extra network hop. Worth having. Different problem.

The limits worth knowing

There's a cheaper answer that people skip past because it feels like a workaround. If your traffic is HTTP and it comes through a layer-7 load balancer or an ingress controller, the client address is already in X-Forwarded-For, and reading it there costs you nothing in distribution. Local earns its keep when you're terminating TCP or TLS yourself, or when you need the address for something below HTTP.

Some clouds also expose a middle path. AWS NLB and the ALB controller can target pod IPs directly rather than NodePorts, which sidesteps the node-level hop entirely and gets you both properties at once. If you're on EKS, check that before reaching for Local.

And a detail worth knowing before you debug this at 3am: pods on a node with Local set can reach the Service through the ClusterIP normally. The policy governs traffic entering from outside. Testing from inside the cluster will not reproduce any of the behaviour described here.

Hyperthreading: the throughput win that costs you latency

Ilia Gusev — Wed, 12 Aug 2026 14:02:09 GMT

Turning hyperthreading off halves your logical core count. Every capacity model you have says that's a bad trade.

Netflix did it on an m7i.metal-24xl and container launch latency improved by 20 to 30%.

The workload was the mount storm from this week's issue: a hundred containers coming up at once, twenty thousand mount syscalls, all funnelling through one global kernel lock. Under that specific shape of load, more logical cores made things worse.

What a hyperthread actually is

An SMT sibling isn't a core. It's a second set of architectural registers bolted onto one physical core, sharing that core's execution units, its L1 and L2 caches, its TLB, and its share of memory bandwidth.

The bet SMT makes is that a single thread leaves a lot of that machinery idle. Waiting on memory, mispredicting a branch, stalling on a dependency. A second thread fills those gaps, and on a mixed workload it usually does, which is where the familiar 20-30% throughput number comes from.

That bet is about throughput. Nobody ever claimed a hyperthread makes an individual thread finish sooner. It can only make it finish later, because the core it was using alone is now shared.

Most of the time you don't care. Then you hit a lock.

Why contention inverts the trade

Take a spinlock, or any lock whose waiters burn CPU rather than sleeping. Two threads on the same physical core, both spinning on the same cache line.

They're now competing twice. Once for the lock, which is the competition you knew about. And once for the execution units and cache of the core they share, which is the one nobody models.

The waiter isn't idle while it spins. It's issuing loads, hammering the same cache line, consuming issue slots. Those are exactly the resources the lock holder needs to finish its critical section and release. So the waiter slows down the holder, which lengthens the critical section, which means more waiting, which means more interference.

Turn SMT off and every runnable thread gets a whole core. Fewer threads spinning at once, and the one holding the lock gets undivided execution resources to get out of the critical section. Total wait drops even though you halved the core count.

The number of logical cores went down. The rate at which the lock changes hands went up. Under contention that second number is the one that decides your latency.

Where this shows up

The pattern needs two things: a hot lock, and enough parallelism to keep both siblings of a core busy fighting over it. That combination is more common than it sounds.

Kubernetes nodes under container churn. CI runners, serverless backends, anything doing rapid pod turnover, where the kernel's mount and cgroup paths become the contended resource. This is the Netflix case.

JVM applications with heavily contended synchronized blocks, or a ConcurrentHashMap where the access pattern collapses onto one bucket. The JVM's biased-lock optimisations stopped helping years ago and were removed in JDK 15.

Databases under high concurrency on one hot table. Postgres buffer-mapping locks, MySQL latch contention. There's a reason database vendors have historically recommended physical-core sizing for these workloads rather than counting vCPUs.

In-memory stores with a global structure. Redis is single-threaded for command execution so it dodges this, but the surrounding I/O threads and anything doing global key-space operations do not.

How to tell whether it's you

The tell is that your tail latency degrades faster than your average as concurrency rises. If p50 is flat and p99 is climbing while CPU utilisation still looks reasonable, contention is a good hypothesis.

From there, perf will tell you where the time goes:

perf lock record -a -- sleep 10
perf lock report --sort wait_total

If a single lock dominates wait_total, you have the first of the two ingredients. Then check whether the busy threads are landing on sibling pairs:

lscpu -e=CPU,CORE,SOCKET | head
# CPU 0 and CPU 24 sharing CORE 0 means those two are siblings

Testing it is cheap and reversible, which is the best thing about this whole topic. You don't need a maintenance window or a new instance type:

echo off | sudo tee /sys/devices/system/cpu/smt/control
# and back
echo on  | sudo tee /sys/devices/system/cpu/smt/control

Run your benchmark both ways on one node. If the answer is no, you've spent ten minutes.

The limits worth knowing

This is not an argument that SMT is bad. For CPU-bound stateless work with little shared state, the throughput win is real and you should take it. Batch processing, video encoding, most stateless request handling that isn't fighting over anything.

The trade also moves with core count. On a machine with a small number of physical cores, halving them hurts more than the contention does, and the inversion may never show up. Netflix measured this on a 24xlarge, where there were plenty of physical cores to go around.

And there's a second reason this button exists. After Spectre, L1TF and the MDS family, several clouds already recommend SMT off for multi-tenant workloads, because sibling threads sharing a core also share the microarchitectural state those attacks read. If you're turning it off for isolation anyway, measure the performance side while you're there. It may not be the cost you budgeted for.