Somebody on my team once sized a node pool off the instance label. c5.4xlarge, 16 vCPU, done, that's what goes in the capacity plan. Then we actually looked at what kubectl describe node reported as allocatable, and it wasn't 16. It was closer to 15.5 on paper and noticeably less than that once the DaemonSets landed. Nobody had lied to us. We'd just never asked the node what it thought "capacity" meant.
That gap, between the number on the instance and the number a pod can actually schedule into, is usually 20-30% on a real cluster, and almost none of it shows up on a kubectl top dashboard. It's not a bug. Every layer taking a slice has a legitimate reason to exist. It's just that nobody sums the slices before doing capacity planning, so the plan is wrong by a fifth to a third before the first workload lands.
Here's where it actually goes.
kube-reserved and system-reserved
Every managed Kubernetes offering carves out CPU and memory for the kubelet, the container runtime, and the OS itself before your pods ever see it. The three big ones do it with three different formulas, and none of them show up on the labels on the instance type.
AKS is the most transparent about the math: kube-reserved is the lesser of 20 MB × max-pods + 50 MB or 25% of total memory. An 8 GB node running 30 pods reserves 650 MB before anything else happens. GKE publishes reference numbers too: a 2 vCPU / 7.5 GB node reserves roughly 70m CPU and 1,736 Mi memory as kube-reserved on its own. EKS doesn't publish an exact formula at all; the common community convention baked into most bootstrap scripts is memory=0.3Gi, ephemeral-storage=1Gi for system-reserved, which is a convention people copied into their tooling, not a number AWS commits to.
Three providers, three different answers to "how much of this node is actually mine," and the honest answer is you have to ask the node, not the instance catalog.
Links
Eviction thresholds, and the version cliff nobody warns you about
Reserved capacity is the visible tax. Eviction thresholds are the one nobody budgets for: they carve out headroom that never shows up as "used" on any graph, because it's specifically the memory the kubelet refuses to let you use, held in reserve for the moment things go sideways.
GKE's default memory.available hard eviction threshold is 100Mi. AKS matches that on 1.29 and later. AKS before 1.29 defaulted to 750Mi, seven and a half times more headroom carved out of every single node, silently, as part of a version bump most teams treat as routine. If you fleet-upgraded across that boundary and your capacity math didn't move, it was wrong the whole time before and it's differently wrong now. EKS's commonly configured value in community bootstrap tooling sits around 200Mi, in between the other two, and again: not a published AWS default, a convention.
None of these thresholds are wrong to have. A node with zero eviction headroom OOM-kills workloads instead of gracefully shedding the least important pod first. The reservation itself is fine. What trips people up is that "allocatable" on the node object already accounts for kube-reserved, while the eviction threshold sits on top of that, invisible to anything reading the allocatable field. Most capacity math stops at allocatable and never goes further.
Links
Pause containers: small, until you're at scale
This one is genuinely tiny per pod: a pause container holding the shared network and IPC namespace for everything else in the pod runs around 1 MB of resident memory. Nobody should reorganize a sprint around it.
But it's 1 MB times every pod on every node, all the time, for the lifetime of the cluster, and it's invisible in the same way the eviction threshold is: real memory the OS has committed, attributed to a container nobody thinks about when they're reading a namespace's usage graph. At 1,000 pods that's roughly a gigabyte of RAM doing nothing but holding namespaces open, spread across your fleet in amounts too small to alert on and too persistent to ever go away.
No budget breaks on it alone. What it does is quietly turn "sum of container requests" into a slightly wrong answer to "how much memory does this cluster actually need" - permanently, by a margin that grows with pod count.
DaemonSets: the capacity your FinOps dashboard calls "infrastructure"
This is the one that actually moves budgets, and it's the one most cost dashboards get structurally wrong.
CNI, CSI, the log shipper, the node-level monitoring agent, whatever security or service-mesh sidecar-injector runs as a DaemonSet - every one of them takes a CPU and memory reservation on every node in the cluster, multiplied by node count, for the entire life of the fleet. This cost scales with your node count, not your workloads, which means it moves in lockstep with your other costs, whatever's already driving the bill.
The reservation itself isn't what costs money. Most FinOps tooling attributes CPU and memory by namespace or by workload label, and DaemonSets frequently live in a kube-system or platform namespace that gets bucketed as "infrastructure" and never allocated back to the teams whose CNI policies, log volume, or security scanning drove the agent's resource needs in the first place - that's where the money actually goes missing. Five or six DaemonSets on a hundred-node cluster is a real, attributable cost that shows up on nobody's per-team invoice.
If your FinOps dashboard has a line called "infrastructure" that's bigger than any single team's line, that's usually where it's hiding.
Hosted control plane fees: the one that's actually on your bill
Everything above is invisible capacity tax. This one, at least, is a real line item - it's just inconsistently visible depending on who's cutting your check.
As of this year, EKS and GKE both charge $0.10 per cluster per hour for the control plane while your Kubernetes version is in standard support - about $73/month, though GKE offsets that with a $74.40 monthly credit per billing account that covers exactly one zonal or Autopilot cluster. AKS's free tier charges nothing for the control plane at all; the $0.10/hour tier buys you a 99.95% uptime SLA you don't get for free. And EKS has a second number worth knowing before you need it: once a cluster's Kubernetes version rolls into extended support, that $0.10/hour becomes $0.60/hour - six times the base rate, for the exact same control plane, as the penalty for not upgrading.
On bare metal, none of this exists as a line item, because the control plane is your hardware and your amortization schedule, which is precisely why it's easy to under-budget for it when you migrate the other direction.
Links
What to actually do with this
kubectl top was never lying to you about usage. It was answering a question you weren't asking - "how much is running" instead of "how much of this node's sticker capacity did I actually get to use." Those are different numbers, and the gap between them is kube-reserved plus the eviction threshold plus pause-container overhead plus your DaemonSet count, none of which show up in the same place.
The concrete version: read kubectl describe node and compare Capacity to Allocatable yourself, on your actual node type, on your actual provider - don't trust the instance catalog. Then subtract your DaemonSet requests times node count before you tell finance what a node pool costs per unit of real workload capacity. That number, not the instance label, is the one your capacity plan should be built on.
Questions? Feedback? Reply to this email. I actually read them.
- Ilia