Capsule's API description says a tenant quota's hard limit "is never crossed." The controller behind it sums usage on a reconcile loop and rewrites each namespace's limit to whatever headroom is left, so between two passes every namespace sees the same leftover. I only found that because I stopped at the sentence and opened the code.
It happened again in all four blocks this week. Grab bag, round three.
🚀 Sandbox Watch: HolmesGPT
What it is
An open-source agent for incident investigation. You ask "why is ledger crashlooping?", it queries your cluster and observability stack, then writes up a root cause. Robusta started it in May 2024 and donated it to the CNCF, where the Sandbox vote passed on October 8, 2025. The repo moved from robusta-dev to its own HolmesGPT org. Apache 2.0, about 3,350 stars, release 0.41.0 on September 8.
How it works
An agentic loop, where the model reads each tool result before choosing its next call. Built-in toolsets reach Kubernetes, Prometheus, Loki, Tempo, Datadog, ArgoCD and dozens more, plus MCP servers. Runbooks are now called skills, markdown procedures synced from git. Models go through LiteLLM, with OpenAI as the default and Anthropic, Bedrock or Ollama a flag away. It ships as a CLI, a Helm-deployed HTTP API, a k9s plugin, or a Slack bot via Robusta.
The honest part
"By design, HolmesGPT has read-only access," says the README. I went looking for where that gets enforced. The Kubernetes toolset skips secrets, and the Helm chart's ClusterRole grants none. A bash toolset ships enabled too, though, with "kubectl get" in its core allowlist and an empty default deny list. On the CLI it runs under your kubeconfig, so a secret read passes whenever your RBAC permits it. Anything off the list stops at a "Do you want to proceed?" prompt, which --bash-always-allow removes. Since February a Remediation MCP server can restart, scale, drain and patch, each change human-approved. That's the policy question from issue #034, inside an SRE tool.
Every log line the loop pulls lands in the model's context, off your network with a hosted provider. Ollama support is labeled experimental, with tool calling that "may produce inconsistent results."
Accuracy, from the project's own August 5 benchmark: seven models passed between 71% and 94% of 63 runs, and five scored 50% on tests tagged "hard". A wrong answer prints under the same "AI:" header as a right one.
Where I'd use it
k8sgpt, Sandbox since December 2023, finds problems with Go analyzers and calls an LLM only for --explain. HolmesGPT lets the model choose what to query, so it reaches further and guesses more. I'd take it for the first ten minutes at 3 a.m., read its conclusion as a list of places to look, and keep --bash-always-allow away from any admin kubeconfig.
Links
💎 The Hidden Gem: backoffLimitPerIndex
What it is
A retry budget per index for Indexed Jobs. backoffLimitPerIndex and maxFailedIndexes went GA in Kubernetes 1.33 and have been on by default since 1.29. Around them the Job API kept growing: podFailurePolicy reached GA in 1.31, successPolicy in 1.33, podReplacementPolicy in 1.34, managedBy in 1.35. The retry knob most people remember is still the original one, backoffLimit, default 6.
One budget for a thousand shards
Picture a 1000-shard Indexed Job with that setting. The controller compares the 6 against status.failed for the whole Job, a counter that doesn't reset when other shards succeed, and fails the Job on the seventh failed pod from any index. Three crashes on shard 7 plus four pods caught in a node drain will do it. Running pods get deleted, unstarted indexes never get a pod, and nothing retries the Job afterwards.
Drains count immediately. Without a pod failure policy, a pod is treated as failed the moment it gets a deletionTimestamp, even if it would have exited 0 inside its grace period.
The docs say retry backoff (10s, doubling) caps at six minutes. I checked pkg/controller/job, and the cap constant has been 10 * time.Minute since 1.28.
What per-index changes
Each index gets its own failure counter and its own backoff clock. A shard that burns through its budget lands in status.failedIndexes while the other 999 keep going. Once failed indexes go past maxFailedIndexes, the controller terminates the Job. Under that cap it still ends Failed, reason FailedIndexes, but with 997 shards done and three index numbers to rerun.
An Ignore rule on the DisruptionTarget pod condition keeps evictions and preemptions off the count, and FailIndex gives up on an index at once on an exit code reserved for bad input.
apiVersion: batch/v1
kind: Job
metadata:
name: reindex-shards
spec:
completionMode: Indexed
completions: 1000
parallelism: 50
# no backoffLimit here: with backoffLimitPerIndex set it defaults to 2147483647
backoffLimitPerIndex: 2 # third failed pod marks this index failed
maxFailedIndexes: 20 # 21st failed index terminates the Job
podFailurePolicy:
rules:
- action: Ignore # drain, preemption, taint eviction, node pressure
onPodConditions:
- type: DisruptionTarget
- action: FailIndex # bad input for this shard, retrying won't help
onExitCodes:
containerName: worker
operator: In
values: [42]
template:
spec:
restartPolicy: Never # required by both backoffLimitPerIndex and podFailurePolicy
containers:
- name: worker
image: registry.example.com/reindex:1.8.2
args: ["--shard=$(JOB_COMPLETION_INDEX)"]Where it still bites
Leftover backoffLimit in the manifest still wins. The controller checks the Job-wide limit first, so a chart that templates backoffLimit: 6 onto every Job brings the shared budget back.
Ignoring DisruptionTarget also swallows kubelet node-pressure evictions. If a shard pushes its node into memory pressure, it gets rescheduled with no retry limit unless activeDeadlineSeconds is set. An OOM kill against the container's own memory limit still counts.
kubectl describe job prints Completed Indexes and nothing for failed ones; those live only in status.failedIndexes.
Version lag is small. The oldest EKS release still in extended support is 1.31, and upstream every field above is on by default there. managedBy is the exception, alpha and off until 1.32.
Links
⚔️ The Showdown: Only one of these tenancy tools changes your namespace count
The setup
I went to pull the Hierarchical Namespace Controller into this comparison and found a read-only repo under kubernetes-retired. SIG Auth voted in February 2025 to archive it "due to a lack of maintainers and adopters".
How each one draws the line
Accurate copies objects down a tree. Annotate a Secret or Role with accurate.cybozu.com/propagate and it lands in the sub-namespaces below. Budgets are out of scope. The sample config even lists ResourceQuota as a propagatable kind, and a copied quota gives each child the full limit. v1.9.0 in June was mostly Renovate bumps. 79 stars.
Capsule is the governance layer: a Tenant owns a capped set of namespaces and pins allowed registries and storage classes. It has sat in CNCF Sandbox since December 2022 and has 2,175 stars; v0.14.5 shipped last Friday. The tenant quota's API description promises the hard limit "is never crossed", so I read the controller. The legacy spec.resourceQuotas is a reconcile loop that sums usage and rewrites each namespace's hard to the leftover headroom. Between passes, every namespace sees the same leftover. v0.14.0 (August 21) added GlobalResourceQuota, which reserves usage atomically at admission, and the old quota system is now deprecated.
With vCluster, the tenant stops being a namespace. Each one gets its own API server, OSS under Apache 2.0. Embedded etcd and Private Nodes need the Free tier, which validates its license against vCluster Platform. v0.37.1 landed yesterday, 11,298 stars.
The #023 cost angle
In #023 the bill scaled with namespace count. Capsule and Accurate leave that count where it was, and Accurate inflates the object count on top: one Secret propagated over 40 sub-namespaces is 41 Secrets in every cluster-wide Secret informer.
vCluster moves the line. Tenant namespaces and CRDs stay in the virtual control plane; the syncer pushes pods, services, endpoints and PVCs, plus only the ConfigMaps and Secrets pods reference, into one host namespace per tenant.
The honest part: pods still run on host nodes, so pod-watching DaemonSets see every one of them. And the bill moves into control planes. The chart default asks for 200m CPU and 256Mi, but the sizing guide recommends 4 CPU and 8 GiB per replica, three replicas, for production. Fifty tenants at that profile is 600 CPU in limits.
My read
Trusted teams sharing Secrets and RBAC
Accurate. Opt-in copying, nothing more.
Internal platform with real per-team budgets
Capsule 0.14+, shared limits in GlobalResourceQuota.
Namespace count already hurting, or tenants need their own CRDs
vCluster, after pricing the control planes.
Still on HNC
Archived since April 2025. The archive thread points at Accurate.
Links
👮 The Policy: IfNotPresent let pod B run an image it had no right to pull
Pod B never showed a credential
Pod A in team-a runs registry.example.com/private/app:1.4.2 with an imagePullSecret. Later the scheduler drops pod B from team-b onto the same node - same image, no imagePullSecrets, default IfNotPresent. The kubelet asks the runtime whether the image is on disk, hears yes, and logs "Container image ... already present on machine". No registry call, so no credential check. kubernetes/kubernetes#18787 reported this in December 2015 and closed in April 2025.
The archive note behind this block forced Always through a Kyverno ClusterPolicy and promised "no limitations." Neither half holds up in 2026.
What the fixes cost
AlwaysPullImages. An admission plugin, off by default, that sets Always on every Pod and rejects anything else. For an unchanged image the pull is a manifest check measured in kilobytes, per KEP-2535. The bill is availability: the kubelet re-checks on every container start, CrashLoopBackOff restarts included, so a registry outage stops pods whose bytes already sit on the node.
KEP-2535, inside the kubelet. The KubeletEnsureSecretPulledImages gate went alpha in v1.33 and flipped to beta, on by default, in v1.35. v1.37 still ships it as beta. Every kubelet pull leaves a record under /var/lib/kubelet/image_manager/ with the secret's namespace, name, UID and credential hash, kept across restarts. Pod B brings no matching credential, so the kubelet pulls for real and the private registry refuses it. A pod reusing a known secret skips the registry.
The policy kind is on its way out too: Kyverno 1.19 deprecated ClusterPolicy on August 20, and 1.20 removes it.
The manifests
For 1.35+ nodes whose kubelet config you own:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# NeverVerify | NeverVerifyPreloadedImages (default)
# NeverVerifyAllowlistedImages | AlwaysVerify
imagePullCredentialsVerificationPolicy: NeverVerifyAllowlistedImages
preloadedImagesVerificationAllowlist:
- registry.k8s.io/* # public only: allowlisted images skip verification entirelyFor nodes you can't configure, or older than 1.35, force Always for the private prefix and nothing else. MutatingAdmissionPolicy went GA in v1.36, so there's no webhook to run:
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicy
metadata:
name: private-images-always-pull
spec:
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
failurePolicy: Fail
reinvocationPolicy: IfNeeded
mutations:
- patchType: ApplyConfiguration
applyConfiguration:
expression: >
Object{
spec: Object.spec{
containers: object.spec.containers
.filter(c, c.image.startsWith("registry.example.com/"))
.map(c, Object.spec.containers{name: c.name, imagePullPolicy: "Always"})
}
}
- patchType: ApplyConfiguration
applyConfiguration:
expression: >
Object{
spec: Object.spec{
initContainers: object.spec.?initContainers.orValue([])
.filter(c, c.image.startsWith("registry.example.com/"))
.map(c, Object.spec.initContainers{name: c.name, imagePullPolicy: "Always"})
}
}
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingAdmissionPolicyBinding
metadata:
name: private-images-always-pull
spec:
policyName: private-images-always-pullI ran it against a v1.37.0 kube-apiserver: both private containers came back Always, and an nginx:1.27 sidecar kept IfNotPresent. Kyverno 1.18+ takes the same mutations block in a policies.kyverno.io/v1 MutatingPolicy.
What stays open
Under the default policy, any image the kubelet didn't pull itself counts as preloaded and stays open to every pod on the node. That includes crictl-based warmers like the DaemonSet in #020, and everything already on disk when the gate first turned on. An in-place upgrade from 1.34 leaves a node full of those. AlwaysVerify closes the gap and puts the registry back in front of them.
GKE's node system config has no field for this policy; EKS AL2023 nodes take raw KubeletConfiguration through nodeadm. Records don't expire either: revoke a registry token and pods referencing that same secret keep starting the cached image until image GC removes it.
Links
Issue #18787 went up on GitHub in December 2015 and was closed in April 2025. The default that shuts the gap shipped with v1.35 that December, ten years after the report. It covers images the kubelet pulled itself, so a crictl preloader like the one in #020 still leaves pod B a way in.
Questions? Feedback? Reply to this email. I actually read them.
Ilia