Every five minutes our tenant-operator logs filled up with "the server has received too many requests and has asked us to try again later". Argo CD refreshes that normally took two seconds stretched to eleven. I ran kubectl get pods -n platform against the same cluster (1.36, three apiserver replicas) in the middle of one of those bursts and it came back in under a second. Apiserver CPU sat at 38%, etcd p99 at 9 ms.
The operator reconciles 40,000 TenantBinding objects spread over 1,800 tenant namespaces. A few weeks earlier someone had added a drift sweep: every five minutes, one goroutine per namespace, each doing an uncached LIST through controller-runtime's API reader. Client QPS had been raised to 1000 for a load test and never lowered. On every five-minute mark about 1,800 LISTs landed within a second or two.
Where the 600 seats go
kube-apiserver budgets concurrency in seats. Each replica gets --max-requests-inflight (400) plus --max-mutating-requests-inflight (200), so 600 seats per replica, and API Priority and Fairness divides them between priority levels in proportion to nominalConcurrencyShares. The objects shipped in 1.36 and 1.37 add up to 245 shares:
workload-low: 100 shares, 245 seats
workload-high, node-high: 40 shares, 98 seats each
system: 30 shares, 74 seats
global-default: 20 shares, 49 seats
leader-election and catch-all: 25 and 13 seats; exempt bypasses the count
A FlowSchema decides which level a request lands in. Our operator runs as a service account in the platform namespace, so it matched the service-accounts FlowSchema (matchingPrecedence 9000) and went to workload-low, next to Argo CD and every other service account outside kube-system. My kubectl identity is a human user. It fell through to global-default at 9900 and never touched workload-low's seats.
workload-low also ships with lendablePercent 90. While it's quiet, other levels can borrow 221 of its 245 seats, leaving a floor of 24, and the borrowing controller only re-divides every 10 seconds.
Links
The dump
The apiserver serves its APF state, per replica, on an unlisted debug path. I trimmed the columns:
$ kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels \
| grep -E 'Name|global|workload' | cut -d, -f1,5,6,8,10 | sed 's/Requests//g' | column -t
PriorityLevelName, Waiting, Executing, Rejected, Cancelled
global-default, 0, 2, 0, 0
workload-high, 0, 11, 0, 0
workload-low, 71, 24, 18406, 212
$ kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1' \
| awk -F', *' '/tenant-operator/ {print $1, $7, $12, $14}' | head -2
workload-low 10 list tenant-a0412
workload-low 10 list tenant-b0077
$ kubectl get tenantbindings -n tenant-a0412 --no-headers | wc -l
22Twenty-four requests executing in a level with 245 seats. The second column of that awk output is InitialSeats: a LIST returning 22 objects was holding 10 seats. One oddity if you read this dump yourself: the TimedoutRequests column is never incremented anywhere in the current queueset.go, so queue time-outs show up as Rejected plus Cancelled.
Links
Why 22 objects cost 10 seats
Since 1.34 the SizeBasedListCostEstimate feature gate (beta, on by default) prices a LIST at one seat per 100 KB the apiserver expects to hold in memory. The estimate takes the object count and average size of the whole resource, whatever namespace the URL names. 40,000 TenantBindings at about 9 KB each is 360 MB. When the estimator assumes the watch cache serves the LIST, it caps memory at 1 MB, so the charge tops out at 10 seats, and every namespace-scoped LIST we sent paid that maximum.
LISTs it expects to hit etcd, a continue-token page for example, skip the 1 MB cap and stop at a per-level limit instead: the smaller of 15% of nominal seats and nominal seats divided by handSize. In workload-low that comes to 37. The kubernetes.io page still describes seats as proportional to object count, and a comment in apf_controller.go says the estimator "limits max seats at 10", while request/config.go raised the LIST ceiling to 100 in July 2025.
Seat width also stalls dispatch. The dispatcher takes the head of the queue with the smallest virtual finish time, and if that head needs 10 seats while 6 are free, nothing dispatches. All 1,800 LISTs were one flow (service-accounts distinguishes by user), shuffle-sharded onto 6 of 128 queues with 50 slots each. Roughly 1,400 were rejected on arrival. The rest could wait a quarter of their deadline, 15 seconds under the default 60-second timeout. Retry-After starts at 1 second and doubles while drops continue, up to 32, and client-go retries a 429 up to 10 times, so one sweep kept coming back for minutes.
Argo CD hashed to other queues and never got a 429. It waited behind 10-seat heads, which is where the eleven-second refreshes came from.
Links
What we changed
The sweep moved to the manager's cached client that same afternoon. It reads TenantBindings from the informer the operator already had, so the five-minute burst stopped reaching the apiserver. QPS went back to defaults.
Then we fenced the operator's LISTs into their own level, so the next uncached read hits a smaller wall:
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
name: tenant-sweep
spec:
type: Limited
limited:
nominalConcurrencyShares: 20
lendablePercent: 50
borrowingLimitPercent: 0
limitResponse:
type: Queue
queuing:
queues: 8
handSize: 2
queueLengthLimit: 50
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
name: tenant-operator-lists
spec:
priorityLevelConfiguration:
name: tenant-sweep
matchingPrecedence: 1000
distinguisherMethod:
type: ByUser
rules:
- subjects:
- kind: ServiceAccount
serviceAccount:
name: tenant-operator
namespace: platform
resourceRules:
- verbs: ["list"]
apiGroups: ["platform.example.com"]
resources: ["tenantbindings"]
namespaces: ["*"]
clusterScope: trueWith 20 more shares the total becomes 265: tenant-sweep gets 46 seats per replica and workload-low drops to 227. Leaving borrowingLimitPercent out would have meant unlimited borrowing, which defeats the fence. The per-request cap follows the level size too, so a LIST in tenant-sweep is charged at most 7 seats. The operator's writes and watches still match service-accounts. We also added alerts on apiserver_flowcontrol_rejected_requests_total by priority_level and reason, and on p99 of apiserver_flowcontrol_request_wait_duration_seconds.
Links
Twenty-three days later
Rejections in workload-low have stayed at zero since the change. tenant-sweep sees a few dozen LISTs a day from a startup check that still uses the API reader, and none of them has queued. The drift sweep finishes in 1.9 seconds from memory, and Argo CD refreshes at :00 and :05 look like the ones at :03. The first fix proposed in the incident channel was raising --max-requests-inflight to 600. That would have handed workload-low 82 more seats, against a sweep that asked for 1,800 requests at 10 seats each every five minutes.
Questions? Feedback? Reply to this email. I actually read them.
- Ilia