This week I mirrored registry.k8s.io/pause:3.10 into a local registry with skopeo. The plain copy tagged a single platform and the upstream index digest was gone. Adding --all brought the index back under a different digest, because skopeo gzipped two uncompressed Windows layers on the way out. The digest only survived with --preserve-digests, a flag that is off unless you pass it.
The other three blocks turned on defaults too. Grab bag, round four.
📑 The KEP Read: Gang scheduling is in kube-scheduler now, switched off
What changed in v1.37
On September 9 we published a Kueue piece that credited the default scheduler with placing pods "one at a time, as capacity allows." Kubernetes 1.37 had shipped gang scheduling inside kube-scheduler as beta 14 days earlier, after an alpha in 1.35. I missed it.
KEP-4671 promoted Workload and PodGroup to scheduling.k8s.io/v1beta1, and the old GangScheduling and WorkloadAwarePreemption gates were folded into GenericWorkload. Shared DRA claims for a PodGroup went beta too, and CompositePodGroup (KEP-6012) arrived as alpha for groups of groups.
Beta does not mean enabled. GenericWorkload defaults to false, and v1beta1 is on the apiserver's disabled-by-default list. I started a v1.37.0 kube-apiserver locally. With only the gate, podgroups wasn't served; with only the runtime-config flag, the log said "PodGroup storage is disabled". It took both, plus the gate on kube-scheduler and kube-controller-manager.
How a gang gets placed now
apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
name: trainer
spec:
schedulingPolicy:
gang:
minCount: 8
disruptionMode:
all: {}
---
apiVersion: v1
kind: Pod
metadata:
name: trainer-0
spec:
schedulingGroup:
podGroupName: trainer
restartPolicy: Never
containers:
- name: worker
image: registry.example.com/trainer:2.4
resources:
limits:
nvidia.com/gpu: 1minCount is the only gang knob and has been mutable since 1.37. There is no timeout field. The scheduler holds member pods until at least minCount of them exist, then evaluates the group against one cluster snapshot. When 7 of 8 fit, none bind. Workload-aware preemption gets a try, then the pods return to the unschedulable queue and PodGroupInitiallyScheduled turns False with reason Unschedulable.
What it doesn't do and where Kueue still sits
The KEP lists fairness and multiple queues as a non-goal: "Kueue and Volcano.sh will continue to provide this." Quota and admission stay in Kueue. So does waitForPodsReady, since the scheduler never updates a group's status after placement, even when pods crash or get evicted.
The only built-in controller that creates PodGroups is Job, behind the alpha WorkloadWithJob gate. Kueue support for the in-tree PodGroup is an open umbrella issue, #8871, with nothing shipped as of v0.20.0-rc.0.
The honest part
Beta still churns. In 1.37 spec.podGroupTemplateRef became spec.workloadRef in types.go (PR #140080), though the PodGroup concept page still shows the old name. The changelog also says to delete v1alpha2 objects before upgrading from 1.36.
Managed clusters are mostly locked out. GKE Rapid has had 1.37 since September 2, but GKE documents switching off-by-default beta gates only for alpha clusters, which get deleted after 30 days. EKS standard support ends at 1.36. AKS release notes from September 4 announce a 1.37 preview rollout and say nothing about workload-aware scheduling.
So the Kueue piece needs a qualifier: "one at a time" still holds on every production managed cluster and on any 1.37 cluster where nobody set the gate. It also quoted Kueue v0.13.4, a September 2025 release. The current one is v0.19.4.
Links
kubernetes-sigs/kueue #8871: Integration with WAS in kube-scheduler
Podo Stack: The scheduler that starts 7 of your 8 GPU pods, then bills you for all 8
💎 The Hidden Gem: pvc-autoresizer
What it is
A small controller from the TopoLVM project (the LVM CSI driver that started in Cybozu's cybozu-go GitHub org) that edits PVCs when their filesystems run low. It never calls a cloud API: it raises the storage request and leaves the expansion to the CSI driver. Apache-2.0, 411 stars, v0.21.1 out on September 2, Kubernetes 1.33 through 1.35 in the README.
How it decides
Each loop (1 minute in the binary, 10 seconds in the Helm chart) it asks Prometheus for kubelet_volume_stats_available_bytes and its capacity twin, plus free and total inodes. When free space or free inodes drop under their thresholds, the request grows by the increase, rounded up to a whole GiB. Both default to 10%. The README says a percentage increase is based on the spec request; internal/runners/pvc_autoresizer.go uses status.capacity.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3-autoresize
annotations:
resize.topolvm.io/enabled: "true" # opt-in per StorageClass
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true # admission rejects PVC resizes without it
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-ingest-0
annotations:
resize.topolvm.io/storage_limit: 2Ti # required: no limit, PVC skipped without a log line
resize.topolvm.io/threshold: 15% # of filesystem capacity (default 10%)
resize.topolvm.io/increase: 20% # of status.capacity (default 10%)
spec:
accessModes: [ReadWriteOnce]
volumeMode: Filesystem # Block volumes are ignored
storageClassName: gp3-autoresize
resources:
requests:
storage: 1TiWhere it waits
AWS dropped the six-hour EBS cooldown on January 15. A volume now gets four modifications in a rolling 24 hours, each after the previous one reaches completed. The extra space appears within seconds, yet AWS's docs say a 1 TiB volume can take up to six hours to finish. When we replayed a log flood against a 1 TiB gp3 claim, the first resize landed in under a minute. Forty minutes later the second bounced off the EBS CSI driver with "in OPTIMIZING state, cannot currently modify", and the disk hit 100% while external-resizer retried.
pvc-autoresizer doesn't stack requests meanwhile. It stores the filesystem size in resize.topolvm.io/pre_capacity_bytes and waits for kubelet to report a new one (a maintainer walked through this in issue #296). A resize that never completes leaves it logging "waiting for resizing..." for good.
Scrape gaps pass unnoticed: a stale kubelet series drops the PVC from the query, and pvcautoresizer_failed_resize_total stays flat.
GCP runs its own clocks, two size changes per 4 hours on Hyperdisk Balanced and one per 6 on Extreme PD. No major cloud shrinks a disk, and Kubernetes rejects a request below status.capacity, so storage_limit is what a runaway log gets to spend and keep.
Nothing native
Kubernetes has nothing native. VolumeAttributesClass (GA in 1.34) changes IOPS or type, and on EBS it spends the same four-a-day budget. I found no KEP for volume autoscaling, and DevOps-Nirvana's Kubernetes-Volume-Autoscaler last released in October 2023. I'd leave the Metrics API mode off: the README points at metrics-server, but the code reads each kubelet through nodes/proxy, the permission from issue #033.
Links
⚔️ The Showdown: Mirroring the pause image with skopeo changed its digest twice
The setup
registry.k8s.io/pause:3.10 is a seven-entry index, five Linux images and two Windows ones, digest ee6521f2. I mirrored it into a local registry:3 and asked each copy for that digest.
skopeo 1.24.0 went first. A plain copy runs with --multi-arch=system, and on a linux/arm64 box that meant one child: the mirror's tag pointed at e50b7059, and a pull by ee6521f2 got MANIFEST_UNKNOWN. An amd64 run lands on 7c38f247 instead.
--all brought every platform back and still missed. The Windows images ship uncompressed layers, and docker/docker_image_dest.go in go.podman.io/image tells a registry destination to compress. One 265 MB layer went out as 109 MB of gzip, and the index landed at 37fae201. The man page doesn't mention it. --preserve-digests returned ee6521f2, and so did a laptop rerun whose blob cache had already seen the raw layers.
Indexes and referrers
crane 0.22.1 and regctl 0.11.6 pushed the index byte for byte; cmp found nothing. Each drops to one platform only when handed a platform flag.
For referrers I used regctl's own image. Its referrers flag carried 33 manifests: a cosign bundle on the index, plus two SBOMs, a cosign bundle and a BuildKit attestation on each of eight platform images. registry:3 returned 404 on the Referrers API, which zot from #033 serves natively, so they landed as nine sha256-digest fallback tags. The digest-tags flag handles the older .sig tags pause:3.10 uses. The crane and skopeo copies had none: cmd/crane has no referrers path, and skopeo moves .sig attachments only when use-sigstore-attachments is set in registries.d.
Sync and day-2
skopeo sync takes a repo or a YAML list with tag filters, keeps copy's defaults and exits after one pass.
regclient's regsync runs as a server: interval or cron per entry, a ratelimit block that waits out source pull limits, a backup tag before an overwrite, the same referrers and digestTags switches. crane has no sync; gcrane cp -r walks repositories through Google's listing API.
One caveat on regclient: it is nearly a one-person project. Brandon Mitchell, who maintains the OCI distribution and image specs, has 1,792 commits; the next contributor has two. 1,930 stars; skopeo, part of a CNCF Sandbox project, has 11,231.
My read
Mirror that must keep digests, signatures and SBOMs
regctl with referrers and digest tags; regsync once it runs on a timer.
Promotion step in a pipeline that already has crane
crane copy. Index intact, referrers left behind.
Existing skopeo jobs
--all --preserve-digestson every copy and sync.
Anything pinned by index digest
crane digest on both sides before the rollout.
Links
👮 The Policy: RBAC already blocks escalation, unless you hand out escalate
The check that already runs
On a v1.37.0 test apiserver alice held a ClusterRole that creates Roles and RoleBindings. Her Role granting get and list on Secrets came back with: user "alice" (groups=["dev-team" "system:authenticated"]) is attempting to grant RBAC permissions not currently held.
An archive note behind this block claimed a team lead could do exactly that and blocked it with a Kyverno validate.roles rule, a field Kyverno doesn't have.
That refusal lives in pkg/registry/rbac/role/policybased/storage.go, with a twin for bindings, and runs before admission. Its "you already hold it" half reads RBAC objects only, which the docs never mention. On GKE that means IAM-granted permissions don't count.
Three verbs and a group that turn it off
escalate on roles or clusterroles skips the check on their contents, aggregationRule guard included. Our ci-deployer service account, holding escalate and bind in one namespace, wrote a secret-reader Role, bound it to itself, and can-i on secrets flipped from no to yes.
bind on a role lets you bind it without holding what it grants.
impersonate borrows an identity instead. Built-in edit carries it for serviceaccounts, and admin inherits it. Constrained impersonation (KEP-5284, default-on beta since v1.36) adds verbs like
impersonate:serviceaccountthat a literal match misses.system:authenticated as a subject. The check never reads the subject list: alice bound her own role to that group, and bob could create ClusterRoles a second later. On GKE that means any Google account; Orca's January 2024 "Sys:All" scan flagged 1,300 of 250,000 clusters.
rbac-tool who-can escalate clusterroles
kubectl auth can-i --list -n payments --as=system:serviceaccount:payments:ci-deployerThe manifest
Swap platform-admins for your group. On the same apiserver it denied every case above, * verbs on the RBAC group and an aggregationRule, and let platform-admins through. Without the aggregation-controller exemption, edit and admin stop picking up aggregated rules, and edit already contains impersonate.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: rbac-escalation-guard
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["rbac.authorization.k8s.io"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["roles", "clusterroles", "rolebindings", "clusterrolebindings"]
matchConditions:
- name: not-platform # masters = apiserver bootstrap, the SA = admin/edit/view aggregation
expression: >-
!request.userInfo.groups.exists(g, g in ["system:masters", "platform-admins"]) &&
request.userInfo.username !=
"system:serviceaccount:kube-system:clusterrole-aggregation-controller"
variables:
- name: riskyRules
expression: >-
(has(object.rules) && object.rules != null ? object.rules : []).filter(r,
r.verbs.exists(v, v in ["escalate", "bind"] || v.startsWith("impersonate")) ||
("*" in r.verbs && has(r.apiGroups) && has(r.resources) &&
r.apiGroups.exists(g, g in ["*", "", "rbac.authorization.k8s.io"]) &&
r.resources.exists(x, x in ["*", "roles", "clusterroles",
"users", "groups", "serviceaccounts"])))
- name: publicSubjects
expression: >-
(has(object.subjects) ? object.subjects : []).map(s, s.name).filter(n,
n in ["system:authenticated", "system:unauthenticated", "system:anonymous"])
validations:
- expression: size(variables.riskyRules) == 0
reason: Forbidden
messageExpression: >-
"escalate/bind/impersonate/wildcard verbs are platform-only: " +
variables.riskyRules.map(r, r.verbs.join(",")).join("; ")
- expression: "!has(object.aggregationRule) || object.aggregationRule == null"
reason: Forbidden
message: aggregationRule is platform-only
- expression: size(variables.publicSubjects) == 0
reason: Forbidden
messageExpression: '"binding to " + variables.publicSubjects.join(", ") + " is blocked"'
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: rbac-escalation-guard
spec:
policyName: rbac-escalation-guard
validationActions: ["Deny"]What it doesn't stop
New grants are all it stops. An identity already holding escalate still writes any Role - ci-deployer's secret-reader went through with the policy on, so the who-can list matters more than the YAML.
Crossplane v2.4.0's RBAC manager holds escalate and bind, and Argo CD v3.5.3's application-controller ships with * on everything, so a synced chart carrying a Role with bind gets denied. Their service accounts go into matchConditions by name, as does the CI identity running Helm.
Exempting system:masters keeps apiserver bootstrap working and doubles as break-glass; #026 has the zero-logins rule for that path.
GKE 1.28+ refuses cluster-admin bindings to the three public subjects; every other role still binds. Two opt-in gcloud flags (1.30.1-gke.1283000+) block the rest and leave existing bindings alone.
Links
Kubernetes docs: Privilege escalation prevention and bootstrapping
Source: pkg/registry/rbac/role/policybased/storage.go (v1.37.0)
The pause image in my test registry answers to ee6521f2 now. It took three copies.
Questions? Feedback? Reply to this email. I actually read them.
Ilia