Podo Stack

Firecracker: the minimalism that runs your Lambda function

Ilia Gusev — Wed, 06 May 2026 14:00:29 GMT

Every AWS Lambda function you've ever invoked ran inside a microVM that started in less than 125 milliseconds, used under 5 MB of memory for the VMM itself, and was destroyed when your function returned. That microVM runs on a piece of software called Firecracker, written in Rust, open-sourced in 2018, and now quietly sitting under Lambda, Fargate, Fly.io, Kata Containers, and half the serverless infrastructure that bills you for single-digit milliseconds at a time.

Most engineers have heard the name. Very few have looked at what Firecracker actually is, why it exists, and where it fits in the boundary-choice conversation that now dominates multi-tenant isolation, AI-agent sandbox platforms, and untrusted-code execution at scale.

Here's the full picture.

Subscribe now

The dilemma that didn't have a clean answer

Before 2018, two ways to isolate arbitrary user code:

Containers (Docker, LXC). Fast start, high density, shared kernel with the host. Process-level isolation through namespaces and cgroups. Strong enough for internal workloads, weak for genuinely untrusted code. "Container escape" is a legitimate attack category.

Traditional VMs (QEMU/KVM). Hardware isolation through a hypervisor. Strong security. Slow start, measured in seconds or minutes. Memory overhead in the hundreds of MB per instance.

AWS needed something for Lambda. Thousands of untrusted functions from different tenants per physical host. Millisecond start times. Strict isolation. Nothing on the shelf fit. QEMU was too heavy. Containers weren't strong enough.

Firecracker is the answer AWS built. A VMM that keeps the hardware isolation of a VM but strips out everything that isn't needed for ephemeral stateless workloads.

What got cut

Firecracker is minimalism as a security feature. The things QEMU emulates that Firecracker refuses to:

No USB, no PCI bus, no BIOS, no graphics. A running microVM sees a minimal virtio-net device, a virtio-block device, a serial console, and one keyboard key for reboot. That's the full hardware surface.
No device passthrough. If you need a GPU, Firecracker is the wrong tool. Cloud Hypervisor or QEMU with VFIO handles that.
No live migration, no complex storage features, no snapshots (for a long time, though snapshots were added later).
No Windows guest support. Linux and OSv only.

Everything cut is attack surface removed. A minimal device model is a minimal set of bugs.

The architecture that makes sub-second boot work

Three choices explain the performance:

Rust at the foundation. Memory safety guarantees eliminate a whole class of bugs that plagued C-based hypervisors. Firecracker's CVE list is noticeably shorter than QEMU's for this reason.

API-driven, not CLI-driven. Firecracker exposes a REST API over a Unix socket. You POST a machine configuration, POST a disk image path, POST a kernel image path, then send an InstanceStart action. No process spawning, no command-line parsing, no shell. Orchestrators build against the API directly.

Jailer. A separate binary that sandboxes the Firecracker process itself using cgroups, namespaces, chroot, and seccomp-bpf syscall filters. If a guest escapes the microVM, it lands in a jailed process with minimal privileges. Two layers of isolation, both required.

The boot numbers are the headline:

Startup to running guest code: under 125 ms.
VMM memory overhead: under 5 MB per microVM.
Density on i3.metal: thousands of microVMs per physical host.

Built-in rate limiting for network and block I/O at the VMM layer means noisy-neighbor problems don't propagate across microVMs sharing a host.

Where Firecracker actually runs in production

Four categories, plus the one people forget:

AWS Lambda. The default execution environment. Your function runs inside a Firecracker microVM that was ready before your request landed.
AWS Fargate. The task runtime for ECS and EKS. Firecracker under the hood, presenting a container API.
Fly.io. Entire platform built on Firecracker as the primitive.
Kata Containers. A CNCF-sandbox project that runs standard OCI containers inside lightweight VMs for stronger isolation. Kata supports multiple VMMs; Firecracker is one of the popular backends. If your cluster has runtimeClassName: kata-fc, Firecracker is the boundary.
AI-agent sandboxes and sandbox-as-a-service platforms. E2B, Daytona, Modal, and the emerging category of remote code-execution platforms all use Firecracker-derived microVMs for isolating tool calls from agents or untrusted snippets from customer tenants. Strong boundary plus millisecond start is the combination that makes the category work.

Where Firecracker is the wrong tool

Three categories of workload that Firecracker can't handle:

GPU-dependent workloads. No PCI passthrough, no direct device access. ML inference and training stay on QEMU or bare metal.
Stateful databases. Disks are ephemeral by design. You can configure persistent storage, but you're working against the grain.
Windows or macOS guests. Not supported. Linux guest kernel only, with OSv as the other option for unikernel use cases.

If your use case needs any of these, look at Cloud Hypervisor (also Rust, newer, different design priorities), Kata with QEMU backend, or just QEMU directly.

The comparison that matters for platform decisions

A clean mental model for boundary strength:

Docker container - boundary: shared kernel + namespaces, start: ms, overhead: MB, untrusted-safe: NO
gVisor - boundary: userspace kernel, start: 100s of ms, overhead: tens of MB, untrusted-safe: yes (weak)
Firecracker microVM - boundary: KVM hypervisor, start: ~125 ms, overhead: 5 MB VMM, untrusted-safe: YES (strong)
QEMU VM - boundary: full hypervisor, start: seconds, overhead: 100s of MB, untrusted-safe: YES (strong)

The rows aren't interchangeable. Picking Firecracker over Docker is a conscious trade: stronger isolation for the overhead of a VMM. Picking Firecracker over QEMU is a different trade: less feature surface for faster boot and lower overhead.

For your own platform, the decision usually comes down to three questions:

Is the workload trusted or untrusted? (Trusted → Docker. Untrusted → microVM or VM.)
Does it need hardware passthrough or Windows? (Yes → QEMU. No → Firecracker.)
Does it need to start in under a second at scale? (Yes → Firecracker. No → QEMU is fine.)

Those three cover most real decisions.

The operational pieces that aren't in the Getting Started docs

If you stand up Firecracker yourself (outside a managed platform), a few things matter:

Kernel choice for the guest. Firecracker documents a minimal Linux kernel config. Running a distro kernel inside works but wastes boot time. The "alpine-microvm" or custom-built minimal kernel is the right choice.
Jailer is not optional. Running Firecracker without the jailer is a production mistake. Every serious deployment runs jailer.
Snapshot/restore (added in later versions). Lets you pre-create a microVM in memory and restore it to a fresh copy in milliseconds. A key primitive for warm-pool patterns in FaaS, AI-agent sandbox platforms, and any sandbox-as-a-service that needs sub-second cold start.
Networking model. Firecracker expects a tap device per microVM. At density, this becomes a host-networking concern, not a VMM concern. Common pattern: one physical host with thousands of tap devices bridged through a fast virtual switch.

Summary

Firecracker is what happens when a single use case (secure multi-tenant serverless) forces a complete rewrite of the VMM concept. The result is narrower than QEMU, stronger than a container, fast enough that boot latency stops dominating your worst-case tail.

If you're building a platform that runs untrusted code at scale, Firecracker belongs in your stack. If you're running standard workloads on standard clusters, Kata-with-Firecracker is an option for tenants you don't trust. And if you're watching the AI-agent sandbox and sandbox-as-a-service category, Firecracker-derived microVMs are the primitive that makes the rest of the security model work.

Subscribe now

For the container-runtime context including the Wasm alternative at the other extreme of the boundary axis, see Tools From the Future. For K8s-native runtime options, eBPF Beyond Networking covers Tetragon as a complementary runtime-security primitive.

Podo #016: Kyverno Isn't Security. It's Governance - And You're Using 20% of It.

Ilia Gusev — Tue, 05 May 2026 14:00:48 GMT

Welcome back to Podo Stack. Kyverno shows up in every platform stack these days, and it almost always arrives the same way: some team ships 80 admission policies they copied from a blog post, mostly image-signing and pod-security, and then declares "Kyverno is done." That's the checklist view.

Last week (Podo #015) I walked through how an attacker reads a pod's first 60 seconds and how each step in the pivot chain maps to a defensive control you probably already own. This week is the flip side: the governance engine those controls actually live in, and why most teams are using a fraction of what Kyverno can do.

Subscribe now

Five surfaces, five rule types, one engine.

Here's what's good this week.

Mutating admission is where the real power lives

Most teams reach for Validate when Mutate would be safer.

A Validating policy rejects the request. Developer hits the wall, files a ticket, platform team writes an exception, and three releases later the policy rots into a set of exemptions nobody can reason about. A Mutating policy fixes the request silently. Developer never notices. The policy stays clean.

Patterns that belong in Mutate, not Validate:

Inject resource requests when a Pod omits them, so you stop landing in BestEffort QoS by accident.
Add a default securityContext (runAsNonRoot: true, readOnlyRootFilesystem: true) to legacy charts you don't own.
Strip the service account token (automountServiceAccountToken: false) for workloads that don't call the Kubernetes API. One line of governance closes Step 1 of last week's pivot chain.
Add cost-center labels from namespace annotations. FinOps attribution without involving application teams.
Rewrite image references to pull from an internal registry mirror, so egress auditing lives in one place.

The architectural idea: Validate is the loud gate that stops bad things from entering, Mutate is the quiet correction that makes them acceptable on the way in. Both belong in a platform. Most platforms ship only the loud one.

There's a real warning attached. Mutate can hide misconfiguration. A pod arrives malformed, Kyverno silently fixes it, nobody learns. Pair every non-trivial Mutate with a policy report, and pipe it to an OpenTelemetry span or an audit stream. Without that trail, Mutate becomes a shadow config layer that outlives the engineer who wrote it.

Rule of thumb I use: if the fix is objective and the developer has no legitimate reason to object, Mutate. If there's a judgment call or a domain meaning the platform shouldn't override, Validate.

Generate rules make cross-namespace propagation trivial

Stop reaching for reflector-style third-party tools.

Every platform team eventually has to distribute the same resource across many namespaces. A TLS wildcard cert. A docker-registry pull secret. A default NetworkPolicy baseline. A CA bundle for in-house registries. The usual answers are all wrong:

Helm post-install hooks run once, drift after any change.
Argo CD ApplicationSets get ugly for dozens of tenants.
Reflector (the third-party controller) is one more binary in your supply chain.

Kyverno Generate rule is the clean answer. Declare one source resource, declare a destination selector, Kyverno watches both. On source change, downstream copies update. On namespace create, generate fires immediately. With synchronize: true, it becomes bidirectional: manual edits on a downstream copy get reverted back to source.

Concrete examples worth building:

Docker-registry pull secret in platform-system, copied into every namespace with label pull-private: true.
cert-manager-issued wildcard TLS secret in cert-manager, copied into every tenant namespace that needs ingress.
A baseline NetworkPolicy in policy-templates, instantiated in every new namespace as the default-deny starting point. This alone closes an entire class of lateral-movement attacks from last week's Step 3.

One anti-pattern worth calling out: don't generate ServiceAccounts with broad permissions via Generate rules. Technically it works. It also breaks RBAC reasoning because the identity that holds a privilege was created by a cluster-scope controller rather than declared by the namespace owner. Generate is for data. RBAC still belongs in the hands of the team that owns the namespace.

Cleanup policies plus kor: making orphans visible

Governance is also about what leaves the cluster.

Every cluster accretes orphans. ConfigMaps from deleted Deployments. Secrets from rotated workloads. Services pointing at Pods that are gone. ServiceAccounts with no RoleBindings. Roles that nothing binds to. It's the kind of debt nobody wants to own and everyone pays for.

Two tools that compose cleanly.

Kyverno CleanupPolicy (built-in since 1.10) gives you declarative TTL. "PVC untouched for 30 days, delete it." Scheduled cleanup via cron expression, or on-event triggers. Always run in auditing: true mode for a week before switching to enforce, because the first cleanup policy a team writes is almost always too eager.

kor (github.com/yonahd/kor) is a CLI that finds orphans Kubernetes-native tools miss. It doesn't just list unused ConfigMaps, it finds Secrets that aren't mounted, Services whose selectors match zero Pods, ServiceAccounts that nothing references, ClusterRoles with no bindings. Output in table|json|yaml|csv so it slots into any pipeline.

A workflow that earns its keep:

Run kor all --output json as a scheduled Job in each namespace.
Pipe findings to a Slack channel per team. Namespace owners decide what's safe to delete.
Promote high-confidence categories (abandoned Secrets older than 90 days, Services with no endpoints for 30 days) into Kyverno CleanupPolicy as enforced rules.

Why this matters beyond cost: an orphaned Secret is a credential-rotation blind spot. The pentester's view from last week (Podo #015, Step 1) lists SA tokens as the primary pivot. If nobody knows that Secret exists, nobody rotates it, and the pivot path stays open indefinitely. Hygiene is security.

CronJob has seven failure modes nobody warned you about

The quietest workload in Kubernetes has the loudest silent failures.

CronJob looks simple. You write a cron expression, you get a scheduled Job. No syslog-style notification when it fails. No on-call alert when it stops running. Three layers (CronJob → Job → Pod), each with its own silent-failure mode.

Seven modes, ordered by frequency in real clusters:

concurrencyPolicy: Forbid. Default is Allow, but many charts flip this to Forbid to avoid overlap. If the previous Job is still running when the next schedule fires, the new one is silently skipped. No error, no event, just a missed run.
startingDeadlineSeconds too small. If the controller lags behind (busy apiserver, scheduler backlog) beyond this window, the invocation is skipped. Set it too short, you miss runs under load. Leave it null, it's valid forever. Some charts ship a 100-second default that is a trap on busy clusters.
backoffLimit exhausted. After N Pod failures (default 6), Kubernetes suspends the Job. The next scheduled time creates a fresh Job, but the suspended one lingers in kubectl get jobs as confusing noise, and nobody alerts you that your nightly backup stopped three days ago.
successfulJobsHistoryLimit and failedJobsHistoryLimit. Set to zero, you lose debug artifacts the moment something breaks. Set too high, etcd pays the bill.
Resource constraints, Pod stuck Pending. No Node fits the requests. CronJob itself doesn't time out a Pending Pod. activeDeadlineSeconds on the Job is the watchdog; set it explicitly.
ImagePullBackOff on a stale image reference. Nobody notices that the nightly backup hasn't pulled a fresh image in three weeks because the registry auth rotated. The Job fires, can't pull, backs off, cycles through retries, exhausts backoffLimit, goes silent.
Kyverno-blocked Job. An admission denial returns no retry. The CronJob controller marks the invocation complete without ever creating a Pod. Your policy is doing the right thing and your scheduler is doing the right thing, and together they silently skip a workload.

Vendor-neutral monitoring, because there's no native answer:

kube_cronjob_next_schedule_time from kube-state-metrics. Alert when time() - last_schedule_time > expected_interval * 2.
A heartbeat beacon emitted from inside the Job container to an external endpoint. Independent of Kubernetes state, so you notice when Kubernetes lies to you.
A Kyverno validate rule that rejects CronJobs without an explicit startingDeadlineSeconds and activeDeadlineSeconds, forcing the author to make the decision.

CronJob is the surface where Kyverno governance earns its keep at the architectural layer. The failure modes aren't bugs, they're the K8s API doing exactly what it said in the spec, silently.

The image hygiene pipeline

Seven vendor-neutral tools, one lifecycle.

This could be seven compilation blocks. It's more useful as one pipeline. Each stage has a tool that does one thing well, and the seams between stages are where platforms earn their quality.

Stage 1 - before build. Hadolint lints the Dockerfile statically. Catches anti-patterns (ADD instead of COPY, apt-get without cleanup, wildcard tags) before a single byte of CI compute burns. Runs in pre-commit and in CI, identical rules.

Stage 2 - build validation. Dive walks the layered image and scores efficiency. "Your image is 1.2 GB and 40% is wasted in inter-layer churn" is a blameless ticket nobody fights. Docker Time Machine adds the time axis: how did this image grow across 20 commits, which PR bloated it by 300 MB. dtm registry myapp --last 20 plots the curve. commit-bisect finds the culprit. Small caveat: DTM has 64 stars and slowed down in late 2025, so check repo pulse before baking into your pipeline. If it stalls, Dive alone still earns Stage 2.

Stage 3 - registry operations. Skopeo and Google's crane both do registry ops without a Docker daemon. Skopeo copies, inspects, and deletes across registries. crane reads partial layers, does content addressing, and fits in slim CI containers. Both CRI-agnostic, both safe to run in ephemeral build agents.

Stage 4 - admission (Kyverno VerifyImages). This is the supply-chain gate. Kyverno checks Cosign signatures (keyless flow via OIDC, signatures anchored in the rekor transparency log) and SLSA provenance attestations before allowing a Pod to start. Fail-closed: unsigned image, Pod rejection, human escalation. Podo #005 walked through why you want this, and the Sigstore/SLSA ecosystem has matured since. If you implemented #005 already, this is where Kyverno enforces the artifact you produced then.

Stage 5 - node cleanup. Eraser removes stale images from nodes. Kyverno CleanupPolicy scrubs workloads that use imagePullPolicy: Always on tags that no longer exist in the registry. Together they close the loop on images that otherwise accumulate quietly on every node forever.

The architectural point: this is one pipeline, not seven tools. A team that adopts Dive without Eraser fixes their build efficiency and still ships nodes with 40 GB of dead image layers. A team that enforces VerifyImages without Hadolint blocks a lot of things at admission that should have been caught in CI. The value is in the seams.

Closing

Kyverno as a governance engine, not an admission controller:

Mutate for objective corrections, not just Validate for loud gates.
Generate for cross-namespace propagation, retire the third-party reflectors.
CleanupPolicy plus kor for the orphan debt nobody else owns.
CronJob rules for the failure modes nobody alerts on.
VerifyImages at the end of a pipeline, not as a standalone policy.

Five rule types, one engine. If your platform ships 80 Kyverno policies and they're all Validate plus VerifyImages, you have four surfaces left to explore.

Which rule type did your team reach for last? Most platforms hit Validate and stop. Generate usually clicks next. CleanupPolicy is the one teams discover after a quarter of orphan debt.

Subscribe now

Next week: Postgres on Kubernetes - five places where the control plane and the database fight over the same recovery, and what running CloudNativePG actually buys you.

- Ilia

Server-Side Apply and managedFields: the Kubernetes feature that ends controller fights

Ilia Gusev — Fri, 01 May 2026 14:02:03 GMT

Most Kubernetes users never read the metadata.managedFields block in a YAML dump. It's boilerplate noise at the bottom of kubectl get -o yaml. Until the day ArgoCD and your operator start fighting over the same Deployment, the spec oscillates every five minutes, and the only signal is three-minute-old pods getting recreated nobody asked for. That's the day managedFields matters.

The structure exists because the client-side kubectl apply model had a fundamental flaw: the client held the "last-applied" annotation and the server had no idea who owned what. When two different actors both tried to be authoritative, whichever called apply most recently won. Server-Side Apply (SSA) moved that ownership tracking into the API server itself, per-field, so conflicts are visible, resolvable, and auditable.

Here's the full story.

Subscribe now

What managedFields actually contains

Every object in a cluster running SSA has a managedFields array in its metadata. Each entry describes one "manager" and the fields it owns:

metadata:
  managedFields:
    - manager: "kubectl"
      operation: "Apply"
      apiVersion: "apps/v1"
      fieldsType: "FieldsV1"
      fieldsV1:
        f:spec:
          f:replicas: {}
          f:template:
            f:metadata:
              f:labels:
                f:app: {}
    - manager: "kube-controller-manager"
      operation: "Update"
      fieldsType: "FieldsV1"
      fieldsV1:
        f:status:
          f:availableReplicas: {}
          f:readyReplicas: {}

Three things worth noticing.

First, ownership is per-field, not per-object. kubectl owns spec.replicas and specific labels. kube-controller-manager owns parts of status. Both coexist on the same Deployment without stepping on each other.

Second, the operation field distinguishes Apply (SSA explicit intent) from Update (legacy imperative PUT). The merge algorithm treats them differently.

Third, the field paths use a compact encoding (f: prefix, nested objects, {} for leaf fields). That's the FieldsV1 format. It's verbose in YAML but efficient for the server to diff.

The conflict a client-side apply user never saw

Imagine three actors touching the same Deployment:

A GitOps tool (ArgoCD) reconciles the Deployment from a manifest in git.
An HPA autoscaler updates spec.replicas based on load.
A platform operator adds a sidecar container to spec.template.spec.containers.

Under client-side apply, every one of them sends a full PUT. Last write wins. ArgoCD says "replicas should be 3", HPA says "replicas should be 17", the operator says "add sidecar". Every few minutes the values thrash. You see pods being recreated. Nobody logs anything useful because from each tool's perspective its own apply succeeded.

Under SSA, each actor declares which fields it intends to own:

ArgoCD applies with fieldManager=argocd and declares ownership of everything except spec.replicas.
HPA applies with fieldManager=horizontal-pod-autoscaler and owns only spec.replicas.
The operator applies with fieldManager=sidecar-injector and owns the sidecar container entry.

Now when ArgoCD reconciles, the API server sees ArgoCD doesn't claim spec.replicas, so HPA's value is preserved. When HPA scales up, the rest of the spec is untouched. When the operator injects a sidecar, ArgoCD's spec.template.spec.containers list is merged, not replaced.

This is the feature. Per-field ownership turns three fighting actors into three cooperating ones.

Force, conflict, and what actually happens on collision

When two managers both try to own the same field, SSA raises a conflict. The request fails with a 409 response and a message listing the conflicting paths. The client has three options:

Drop the conflicting field from its apply. The other manager keeps ownership.
Send force=true to take ownership. The other manager loses that field.
Abort and investigate. Usually the right answer when the conflict is surprising.

kubectl apply --server-side --force-conflicts is the escape hatch for admins. Tools like ArgoCD and Flux offer equivalent flags. Using force without understanding the cause of the conflict is how three-way ownership oscillation starts.

The more disciplined pattern: if you repeatedly hit conflicts on a specific field, that's a signal the ownership model is wrong. Two managers shouldn't both be writing to the same field unless one of them is actively handing ownership off.

List merge strategies: where subtle bugs live

Lists are where SSA gets interesting. By default, a list is treated as atomic: the whole list is owned by whoever applied it. But most Kubernetes lists in practice have semantic identity per element (containers by name, ports by number, volumes by name). The API declares merge strategies for each such field using x-kubernetes-list-type and x-kubernetes-list-map-keys markers in the OpenAPI schema.

For example, spec.template.spec.containers is declared as list-type: map with list-map-keys: [name]. When two managers apply to the containers list, SSA merges element-by-element by container name. Manager A can own container "app", manager B can own container "sidecar", both survive.

If a list has the wrong marker, merge breaks. A common failure is a Custom Resource Definition that declares a list without map keys. Two actors apply different elements; the second apply replaces the entire list. The first actor's element silently disappears. You learn about this when something that was there is gone and nobody's manager is listed.

Operationally: when writing a CRD, always think through the list-type markers for every list field. When debugging a vanishing config, look at the OpenAPI schema of the resource and check if the list has proper identity markers.

How ownership transfers happen

Ownership isn't permanent. It transfers when a manager stops claiming a field. Three ways:

A manager applies without the field. If actor A previously owned spec.replicas by applying it, then applies a new manifest that omits spec.replicas, ownership is released. The next actor to apply the field becomes the owner.
Explicit removal. A manager applies with the field present but flagged for removal (via a patch, not SSA directly). Useful for operators that want to clean up after themselves.
Force takeover. Already discussed. force=true reassigns ownership to the caller.

The subtle one is case 1. Many GitOps tools don't realize that adding or removing a line in a manifest changes ownership declaration on next reconcile. If you remove a field from git thinking it'll revert to default, ArgoCD stops claiming it, and whatever default the controller computes is now owned by the controller. Sometimes that's what you want. Sometimes that's a surprise.

The operator pattern that gets SSA right

Operators that coexist well with GitOps and HPAs follow a specific pattern:

Apply only what you own. Don't include fields in your SSA apply that belong to other actors. If your operator manages sidecars, send only the sidecar fields, not the whole spec.template.spec.
Use a unique field manager name. my-operator-v2, not my-operator. This keeps ownership traceable across operator versions.
Fail loudly on conflicts. Don't default to force=true. Surface conflicts as events on the managed resource so cluster operators can see what happened.
Design for shared resources. Assume another actor might own adjacent fields. Don't assume you're alone.

Crossplane (Podo #010) is an example of a controller family that uses SSA this way by default. Karpenter's NodePool reconciliation also follows the pattern, which is part of why NodePool configurations coexist cleanly with kubectl edits.

When you still want client-side apply

SSA isn't free. The server does more work per request. managedFields adds size to every object (sometimes multi-kilobyte blocks). The merge logic has edge cases (especially around map-typed fields with heterogeneous keys).

For one-shot scripts, kubectl admin fiddling, ephemeral test clusters, client-side apply remains simpler. The cost of tracking field ownership across actors is overhead you don't need when there's only one actor.

For anything with more than one source of truth (GitOps plus operator plus HPA, the common production shape), SSA is non-optional. The bugs you'll hit without it are worse than the overhead.

Practical diagnostics

Three commands that pay for themselves:

kubectl get -o yaml | yq '.metadata.managedFields' - shows the full ownership map. Use when something changed and you don't know who changed it.
kubectl get -o json | jq '.metadata.managedFields[] | {manager, operation, fields: .fieldsV1}' - structured view for scripting. Audit pipelines can parse this.
kubectl diff --server-side - shows what SSA will do before you actually apply. Includes conflict warnings. Indispensable for GitOps CI pipelines.

One warning: managedFields can balloon. If dozens of actors with unique field manager names apply to the same resource over a long period, the block grows. Kubernetes 1.24+ has cleanup logic that removes entries for managers that haven't touched the object recently. Earlier clusters sometimes need manual pruning on long-lived resources.

Summary

managedFields is the piece of Kubernetes metadata that makes cooperation between controllers actually work. Per-field ownership, declared intent, explicit conflict surfacing. Every GitOps-meets-operator-meets-HPA production cluster depends on it, usually without realizing.

If you're writing operators, use a unique fieldManager name and apply only what you own. If you're running GitOps, choose a tool that uses SSA natively and set conflict policies consciously. If you're debugging a resource that keeps changing without a clear cause, kubectl get -o yaml and read the managedFields. The answer is usually there.

Subscribe now

For the broader GitOps context where this matters most, see Your Infrastructure Has an API Now. For image pipeline conflicts specifically, Karpenter Beyond the Demo walks through NodePool/EC2NodeClass ownership cleanly.

Prometheus WAL: the silent disk growth nobody diagnoses correctly

Ilia Gusev — Wed, 29 Apr 2026 14:01:52 GMT

You don't think about the write-ahead log until the day your Prometheus takes 22 minutes to start after an OOM kill, or you notice your data/wal directory is 40 GB on a cluster that scrapes "maybe a couple thousand targets", or a dashboard query returns nothing for the last two hours and the only log line is corruption detected. Each of those is a WAL problem with a very specific cause you can diagnose once you know what the thing actually is.

Prometheus buffers new samples in memory (the Head Block) and simultaneously appends them to a journal on disk (the WAL). The in-memory copy is fast to query. The on-disk copy is durable. Between them, they answer "what happens if the process dies" without asking you to run a cluster of synchronously-replicated databases just to collect metrics. The tradeoff is that the journal is now an operational surface with its own failure modes.

Here's the full story.

What the WAL actually contains

The WAL lives at data/wal/ inside your TSDB data directory. Inside, you'll find sequentially numbered files like 000002, 000003, 000004. Each segment is capped at 128 MiB. That size is hardcoded, you can't tune it with a flag.

Inside each segment, three kinds of records:

Raw samples: a timestamp plus a value, tagged with a series reference.
Series descriptors: the label sets for each new time series, with a monotonic ID.
Tombstones: markers for deleted data, created when you call the delete API.

The WAL only holds the active Head Block's data, which is typically the last 2 to 3 hours of scrapes. Once Prometheus compacts older samples into a persistent block, those samples get purged from the WAL. What remains in the WAL is whatever the running Head Block still needs. The journal is a window, not an archive.

Ingestion through the WAL: the write path

When a scrape completes, each new sample goes through three phases. First, the sample gets appended to the current active WAL segment on disk. Second, it gets inserted into the in-memory Head Block so queries can see it immediately. Third, when the active segment crosses 128 MiB, Prometheus creates a new segment and starts writing there.

The order matters. If the process crashes between the WAL append and the Head insert, the next startup will replay the WAL and rebuild the Head. If the crash happens before the WAL append completes, that sample is simply lost, which is the expected tradeoff for a metrics system as opposed to a bank ledger.

Rotation is continuous. A busy Prometheus scraping thousands of targets rotates segments every few minutes. A quiet one might keep the same segment for an hour.

Checkpointing: the mechanism that makes WAL bounded

This is the part most engineers never look at, and it's where the real magic is.

Every two hours (by default, driven by min_block_duration), the Head Block truncates. The oldest samples compact into a persistent block on disk, and the Head now covers a shorter time window. But the WAL still contains records for samples that are now in the persistent block, plus records for series that no longer have any samples in the Head.

To clean this up, Prometheus writes a checkpoint. A checkpoint is essentially a filtered WAL, containing only the records that are still relevant to the active Head:

Series descriptors for series that still have active samples.
Raw samples within the current Head window.
Tombstones that haven't been fully applied yet.

After the checkpoint finishes, Prometheus deletes the old WAL segments. The WAL size drops. The next replay after a crash will read the checkpoint plus any WAL segments written after it.

This is why WAL size is bursty. A healthy Prometheus sawtooths: grows linearly between compactions, drops at each compaction boundary. If you watch data/wal/ over time and see the sawtooth, you're fine. If you see monotonic growth with no drops, something is wrong. Usually the something is cardinality.

Cardinality as a WAL problem, not a memory problem

Most writeups about cardinality focus on RAM. Prometheus running out of memory, query OOMs, slow lookups. That's all true. The less-discussed side is that every new series gets a new series descriptor record in the WAL, and every scrape of every series appends another sample record.

If your services legitimately expose 50 series each and you have 200 services, your WAL grows at a rate determined by 10,000 series times your scrape interval. Predictable and manageable.

If your services expose 50 base series but those series carry a pod_template_hash label and you deploy four times a day, you're actually generating new series every deployment. Each deployment means 10,000 new series descriptor records in the WAL, and then the old 10,000 series keep getting samples until the Head drops them.

"High churn" is the Prometheus jargon for this. Labels that change frequently. Kubernetes pods with hashed names. Kafka consumer groups with generation IDs. Job labels that include the CI build ID. Each of these quietly inflates the WAL because every unique label combination creates a new series, and the series descriptor has to live in the WAL until the checkpoint catches up.

Diagnose churn with one PromQL query:

rate(prometheus_tsdb_head_series_created_total[1h])

Divide by the rate of deletion. If creation runs an order of magnitude faster than deletion, you have churn. The WAL is telling you.

Why your Prometheus took 22 minutes to start

When Prometheus boots, it doesn't trust that memory is correct. The Head Block starts empty. To rebuild it, Prometheus reads the most recent checkpoint, then reads every WAL segment written after that checkpoint, replaying the records in order.

The time this takes is linear in the size of the WAL. A 1 GB WAL replays in maybe 30 seconds on a reasonable disk. A 20 GB WAL (which happens in churn-heavy clusters) can take 20 minutes. Your Prometheus is up but not ingesting until the replay completes. Grafana dashboards dependent on that Prometheus are lying to you the entire time.

Two mitigations, in order of reach:

Enable WAL compression. The flag is --storage.tsdb.wal-compression, on by default since Prometheus 2.20. If you're running an older version, turn it on explicitly. Snappy-compressed WAL reads faster because there's less disk to read. Typical 2-3x size reduction.
Reduce cardinality. Everything in the previous section. Drop the pod_template_hash label, use service-level aggregation, metric_relabel_configs to blacklist high-cardinality labels at scrape time. This is the structural fix.

There's a third mitigation that's an escape hatch, not a solution: delete the WAL directory. Your last 2-3 hours of data go away but Prometheus starts clean. Do this only when the alternative is hours of unavailability and you're willing to accept the data loss.

WAL corruption: what actually happens

Disk failures, filesystem issues, or interrupted writes during a crash can produce a corrupted WAL segment. Startup fails with corruption detected at record N in segment 00000X.

Since Prometheus 2.10 or so, recovery is mostly automatic. Prometheus skips the corrupt record, logs a warning, and continues replay from the next valid record. You lose whatever samples were in the bad record, typically a few seconds of data for a subset of series.

The remaining manual cases:

Prometheus crashes during replay, unable to skip. Delete the bad segment manually (find it by log line), restart. You lose that segment's samples but keep the rest.
A checkpoint is corrupt. Delete the checkpoint file and let Prometheus recompute from the preceding WAL segments.
Everything is corrupt. Delete data/wal/ entirely. Keep your persistent blocks in data/ untouched. You lose the last Head window (2-3 hours), historical data is intact.

The right default is to treat the WAL as recoverable but ephemeral. Never manually edit it. Never try to recover specific records. Either you trust Prometheus to skip and move on, or you wipe and accept the loss.

Disk layout that actually matters

If you're sizing storage, the WAL and the persistent blocks live under the same data/ directory. They grow differently:

WAL: bursty, sawtooth, bounded by Head Block window times churn rate. Rarely over 10 GB on a well-tuned instance.
Persistent blocks: monotonic in retention window times ingestion rate. Can be hundreds of GB or TBs.

If you run Prometheus on a single filesystem, the WAL can sometimes fill the disk before the persistent blocks notice. The symptom is write error: disk full in the logs and all scrapes failing simultaneously. The fix is either to put data/wal/ on a separate volume with its own capacity guarantee, or to monitor free space on the shared volume with an alert well below full.

The three signals to actually alert on

Most Prometheus monitoring setups alert on up{job="prometheus"} == 0. That's fine but it's a lagging signal. Three leading signals that catch WAL issues before they become outages:

prometheus_tsdb_wal_corruptions_total: any non-zero rate means disk or filesystem trouble. Page on a rate greater than zero.
prometheus_tsdb_head_series: plot over time. A linear climb without drops means your checkpoints are not catching up, cardinality is probably exploding. Warn if growth is 2x over 24h.
prometheus_tsdb_head_max_time - prometheus_tsdb_head_min_time: the Head window. Should be roughly min_block_duration. If it drifts wider, compaction is falling behind. Warn if it exceeds 3 hours.

These three capture the main operational failure modes: corruption, cardinality explosion, and compaction lag. None of them are in the default dashboards shipped with kube-prometheus-stack, so you have to add them yourself.

Summary

The WAL isn't complicated once you see the mechanics. A bounded journal, checkpointed every two hours, replayed on startup. The failure modes are all downstream of one root cause: your metric shape. Low cardinality plus compression plus one filesystem with headroom is a WAL that takes care of itself. High churn plus no compression plus a shared volume is a WAL that will wake you up on a weekend.

If you only walk away with one operational change, it's this: turn on WAL compression, alert on prometheus_tsdb_wal_corruptions_total, and put a cardinality SLO in your FinOps conversation. The WAL will stop being interesting, which is what you want.

More on observability cost math in our Prometheus Cardinality FinOps deep dive. For the eBPF-based alternative that measures per-workload resource pressure directly, see eBPF Beyond Networking.

Podo #015: The Quietest Performance Bug: What Happens in a Pod's First 60 Seconds

Ilia Gusev — Tue, 28 Apr 2026 14:01:16 GMT

Welcome back to Podo Stack. Every SRE I know has a graph they can't explain - pods that looked fine at deploy time but started eating P99 latency when the cluster filled up. Most of the time it isn't a memory leak, not GC, not a bad query. It's the first 60 seconds of a pod's life, where four separate things go wrong quietly and no standard dashboard shows any of them.

This week: cgroup v2 weight drift that ate 61% of your CPU priority without telling you, four different ways to beat image pull delay, why the scheduler can't see the noisiest neighbors, and what a pentester sees when a pod lands in your cluster.

Subscribe now

Here's what's good this week.

cgroup v2 silently ate 61% of your CPU priority

The linear formula was the wrong shape.

If you migrated to cgroup v2 recently (modern K8s distros require it now), you probably ran smoke tests, saw pods running, called it done. What you actually shipped, depending on your OCI runtime version, is a cluster where cpu.weight values are silently wrong.

The short story: runc and crun used a linear formula to translate v1 cpu.shares into v2 cpu.weight. Math-wise, that's the wrong shape. v1 shares are on a log scale, v2 weight is bounded on [1, 10000] with default 100. A Pod with 1024 shares (one full CPU) ended up with weight 39, not the expected default of 100. A pod with 100m CPU request got weight 4 - not enough granularity for sub-cgroups to mean anything.

Under contention, that pod behaves as if it has 39% of the priority it thinks it has. You don't see it in kubectl top. You don't see it in node-exporter. You see it in the P99 graphs nobody can explain.

The fix (kubernetes.io/blog/2026/01/30/new-cgroup-v1-to-v2-cpu-conversion-formula/, Itamar Holder at Red Hat) is a quadratic formula in log space:

cpu.weight = ⌈10^(L²/612 + 125L/612 - 7/34)⌉
where L = log₂(shares)

After the fix, 1024 shares map to weight 102 (near the default 100). 100m CPU maps to weight 17, which is actually usable for sub-cgroup arithmetic.

The critical catch: the fix lives in the OCI runtime, not in Kubernetes. runc 1.3.2+ or crun 1.23+. If your nodes run an older containerd that bundles older runc, you still ship the broken formula. GitHub issue kubernetes/kubernetes#131216 tracks the adoption lag.

How to actually check on a running node:

$ cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/.../cpu.weight
39
$ runc --version
runc version 1.1.12

That's weight 39 instead of ~100. Time to upgrade the runtime.

This touches bin-packing too. Karpenter's consolidation (Podo #014) and the FinOps math from Podo #007 both assume cpu.weight reflects the workload's actual priority claim. When weights are wrong by a factor of 2-3x, cost decisions inherit that error. The silent-failure category is the worst kind - everything keeps running, just quieter and worse than it should, forever.

Four ways to kill image pull delay

One problem, four philosophies, and they compose.

Cold image pull is the dominant source of pod startup latency once you've fixed the cgroup problem. You have four options in production, and most teams only know one.

Stargz (Podo #001) is lazy pull. The image is read on demand, so a 2 GB container can start serving traffic within seconds instead of waiting for the full pull. Pros: zero pre-work, cold nodes fast on first touch. Cons: first access to any uncached chunk still hits the registry, and you've added a dependency in the hot path.

Spegel turns nodes into peers. Each node serves image layers it already has to other nodes in the cluster over P2P. Pros: kills registry pressure on burst scale, keeps bandwidth inside the cluster. Cons: every node has to participate, and someone still cold-starts the first copy somewhere.

Image Preload Operator runs as a DaemonSet that pulls a declared set of images ahead of time. Pros: deterministic readiness, CRI-agnostic (containerd, CRI-O, Docker). Cons: fetches images that pods may never schedule, so disk goes to waste on node pools with high tenant variance.

Baked AMI (or golden images) bakes the container image into the node image itself. Pros: literally zero pull at boot, simplest ops surface. Cons: every image update means an AMI rebuild, slow feedback loop, and cross-version fleets get painful.

When to pick what:

Burst capacity, unpredictable image set → Stargz
Steady workload, small stable image set → Baked AMI
Multi-tenant cluster with shared base images → Spegel
Scheduled known workload per node pool → Preload Operator

These aren't mutually exclusive. Stargz + Spegel is a real production combination: Stargz handles cold reads, Spegel shaves registry egress when multiple nodes pull the same layer in the same burst.

What's wrong with picking just one: cold-start tail latency is multimodal. Single-philosophy solutions optimize the median and leave the tail. Layered solutions cost more to run but cap the worst case, which is the case you're actually paying for when a node group scales from zero.

The scheduler can't see the neighbor stealing your latency

Cache contention lives in a blind spot.

The default Kubernetes scheduler scores nodes on CPU requests, memory requests, and a handful of affinity rules. That model assumes workloads are CPU- or memory-bound in ways that show up in requests.

The failure case is micro-architectural contention. A Pod with 20% CPU utilization can thrash the L3 cache hard enough to add 65% to the P99 latency of a neighbor pod. Neither pod violates its limits. Both look healthy in kubectl top.

Metrics absent from requests and limits:

LLC (last-level cache) miss rate
Memory bandwidth saturation (DRAM pressure)
Disk I/O wait at the block layer
NUMA cross-socket traffic

How to collect any of this on a running cluster:

eBPF programs (Tetragon, Parca - Podo #011 covers the stack) hook cache-miss and IPC counters per cgroup.
Intel RDT (Resource Director Technology) surfaces hardware counters for cache occupancy and memory bandwidth per core.
PMU counters via perf are classic but per-node, not per-workload, without extra wiring.

What you can actually do once you have data:

KubeAttention is one example of an ML scheduler plugin that scores nodes on 26 interference features. Not the answer, representative of the category.
Intel CAT (Cache Allocation Technology) partitions L3 cache between cgroup groups at the hardware level, works with cpuset isolation.
Descheduler is the reactive option: it moves pods off nodes where interference crossed a threshold. Fixes the symptom, not the placement.
cpuManagerPolicy: static plus topology manager pins CPUs per pod, avoiding NUMA crossing.

The frustrating part: teams hit this problem, spend weeks blaming the application code, and never notice the scheduler decision was the bug. No metric the default scheduler reads captures cache contention, so the root cause lives outside its field of view.

Interference-aware scheduling sounds exotic until you hit the wall. After that it looks obvious.

A pentester's view of your brand-new pod

The same 60 seconds, read from the other side.

Flip the perspective. You've deployed a workload. Some dependency had a CVE, or a test app was exposed on NodePort by accident, or a supply-chain compromise landed an RCE in a library you vendored last quarter. Either way, an attacker has code execution inside a Pod.

What do they see in the first 60 seconds? The same window we've been staring at, from the other side.

Step 1 - Service Account token./var/run/secrets/kubernetes.io/serviceaccount/token is mounted by default unless you disabled it. kubectl auth can-i --list with that token enumerates the SA's reach. Most workloads carry more RBAC than they need because defaults accrete across Helm chart generations.

Step 2 - RBAC escalation. Typical wins: create pods in kube-system, exec into arbitrary pods, patch clusterrole, bind clusterrole to a new SA. None of these should live in an application SA, and all are common in real clusters.

Step 3 - Kubelet API on neighbor nodes. Port 10250 on every node speaks the Kubelet API. With the right token or relaxed auth, kubeletctl exec's into any pod on that node, reads the secrets mounted there, pivots through the node without ever touching kube-apiserver. Audit logs won't show this path - the apiserver never saw the call.

Step 4 - IMDS.169.254.169.254 is the cloud metadata endpoint. On AWS, it hands over the node's IAM role credentials if IMDSv2 with hop-limit 1 isn't enforced. From there, an AssumeRole chain takes the attacker out of Kubernetes entirely and into your AWS/GCP/Azure account.

Step 5 - hostPath or hostPID escape. If your cluster allows privileged pods anywhere, the attacker mints a new Pod with hostPath: / and reads the kubelet's kubeconfig directly. Now they are the kubelet.

Each step maps to a defensive control most platforms already own:

SA over-reach → Kyverno policy restricting SA permissions, plus explicit automountServiceAccountToken: false where safe
RBAC escalation → audit with rakkess or kubectl-who-can, Kyverno ClusterRole constraints
Kubelet port 10250 → NetworkPolicy blocking the port between namespaces
IMDS → Kyverno policy blocking metadata access from workload pods
hostPath escape → Pod Security Standards at restricted, plus Tetragon runtime policy

The offensive view is what gives the defensive policy corpus its meaning. Most clusters ship 80+ Kyverno policies as a checklist. If you can't trace each one back to an attack step it blocks, you're maintaining config theatre. And config theatre expires - someone refactors a policy for "clarity", the mapping it was guarding gets lost, and you find out three years later during an incident review.

References worth keeping near the desk: CIS Kubernetes Benchmark, NIST SP 800-190, OWASP Kubernetes Top Ten, OWASP Kubernetes Goat (a lab you can practice against in a kind cluster).

Next week's issue is the defensive half of this story - Kyverno beyond admission, the governance layer that makes this policy corpus hold together.

Closing

Demos ship. Production ships. Between those, four silent failures share one trait - they only show up under contention or compromise:

cgroup weight that makes your pod 40% of the priority it thinks it has
image pull stalls nobody attributed to cold start
neighbor pods eating your cache while everyone blames your code
an attacker's pivot chain that starts exactly where your platform stops looking

The first 60 seconds of a pod's life are where demos lie and where platform engineering earns its keep.

Which of these did you find the hard way? Reply and tell me - cgroup drift, image pull tail, noisy neighbor P99, or a compromised SA token. I'm collecting stories for a future issue.

Subscribe now

- Ilia

HDFS Lost. How Object Storage and Table Formats Won the Data Lake

Ilia Gusev — Fri, 24 Apr 2026 14:02:17 GMT

Ten years ago, the conventional wisdom for building a data lake was carved in stone. Data is too big to move, so move the code to the data. Run your Spark jobs on the same machines where your HDFS blocks live. The whole architecture of Hadoop was built on this single principle - data locality - and for a while it was unquestionable.

Today that principle is dead, and data engineering is happier for it. Somewhere between 2018 and 2022, the industry quietly abandoned HDFS for object storage plus ephemeral compute, and the world kept turning. If you joined the field in the last five years, you might have missed the memo that this was once controversial.

So what actually happened? Why did data locality lose? And why is S3, which was supposed to be terrible for analytics, now the default?

The collapse of data locality

When Hadoop was designed, data center networking was 1 Gbps and hard drives were spinning rust. Reading 100 MB from a local disk took about a second. Sending the same 100 MB across the network took eight seconds. Of course you moved the code to the data - the math didn’t leave any other option.

In a modern cloud, the math is completely different. EC2 instances routinely get 25 Gbps of network throughput, with some families hitting 100 Gbps. Meanwhile, a local SATA SSD maxes out at maybe 500 MB/s, and even NVMe tops out at a few GB/s per drive. Network bandwidth has caught up to disk bandwidth and in many cases surpassed it.

The moment network I/O is no longer the bottleneck, the whole reason for data locality disappears. You can read a Parquet file from S3 in the same time you’d read it from local disk. The bottleneck has moved elsewhere - to CPU, specifically to the cost of deserializing columnar data into your query engine’s memory format. “Move code to the data” became “move data to whatever compute is cheapest right now.”

The coupling problem

HDFS doesn’t just assume data locality - it bakes it into the architecture. Your DataNodes are both storage and compute. They hold the blocks and they run the YARN containers. You can’t scale them independently because they’re the same thing.

This creates a painful dilemma every time you want to resize your cluster. Running out of storage? You add nodes, each of which brings CPU and RAM you may not need. Running out of CPU? You add nodes, each of which brings disks you may not need. Your actual resource ratio almost never matches the fixed ratio that your instance type offers, so you’re always overpaying for one thing to get enough of the other.

And you pay 24/7. The nodes have to stay up because they store data. Even during the long stretches when nothing is running, you’re paying full price for compute that’s idle because the disks attached to it have your data.

The object storage model breaks this coupling completely. Your data lives in S3. Your compute lives on EC2 Spot or Kubernetes or a managed service. You scale them independently. You turn compute off when you don’t need it. Your nightly ETL runs on a cluster that exists for 40 minutes and then disappears. You’re paying for seconds of compute, not months of idle storage + compute bundles.

The rename problem and why it didn’t matter

There’s one technical reason people initially thought S3 would never work for serious data lakes. HDFS is a filesystem with a proper tree structure. Operations like rename are atomic metadata operations - rename a directory, flip a pointer, done. Spark’s output committers relied on this: jobs wrote to _temporary/attempt_X/, and on success, renamed the temp directory to the final location in one atomic step.

S3 is not a filesystem. It’s a flat key-value store pretending to have directories. There’s no “rename” operation. Moving an object means copying it to the new key and deleting the old one. Renaming a “directory” with 10,000 objects means copying 10,000 objects - and it’s not atomic, so a failure halfway through leaves you with a mess.

The industry’s first attempts to deal with this were hacks on top of hacks. S3A got “magic committers” that abused S3 multipart upload to simulate atomic commits. It worked but was fragile and required tuning.

The real fix came from somewhere unexpected: the metadata layer. Table formats - Iceberg, Delta Lake, Hudi - solved the rename problem by refusing to use directories as state. Instead of “the data for this table is whatever files are in this directory,” they maintain a manifest of which files belong to the current version of the table. Committing a new version means writing a new manifest and atomically swapping a pointer to it. The underlying object store doesn’t need directories, doesn’t need rename, doesn’t need anything clever. It just stores immutable files and serves them back.

This wasn’t just a workaround. It was an upgrade. Table formats gave you ACID transactions, schema evolution, time travel, hidden partitioning, and efficient deletes - none of which HDFS-based data lakes had.

What won

The stack that replaced HDFS looks like this: S3 (or equivalent) for cheap durable storage, a table format (Iceberg or Delta Lake) for ACID transactions and metadata, and ephemeral compute (Spark, Trino, DuckDB, Snowflake, whatever) that reads directly from S3. Each layer is independent. You can swap engines without touching data. You can run two engines against the same table. Your analyst uses Trino, your ML pipeline uses Spark, your dashboards use DuckDB - all pointing at the same files.

The operational savings are real. No NameNode to babysit. No block balancer to tune. No “we need a Hadoop admin” line in your job requisition. The cloud provider handles durability, replication, and scaling - you pay for what you store and what you read.

When HDFS still makes sense

There’s one case where HDFS is still the right answer: you already own the hardware. If you’ve got a rack full of servers in your own datacenter, each with local drives, you don’t benefit from decoupling storage and compute - you can’t turn the servers off to save money because you’ve already bought them. HDFS extracts the most value from that hardware because it uses both the disks and the CPUs at the same time.

For cloud-based data platforms, running HDFS on EC2 instances is an anti-pattern. You’re paying for compute you can’t turn off, you’re getting none of the benefits the on-prem model offers, and you’re fighting every managed service and ecosystem tool that assumes S3.

The wider lesson

Data locality was not a timeless principle. It was a consequence of a specific hardware ratio - slow networks, slow disks, local CPU - that stopped being true. When the ratio shifted, the architecture built on it stopped making sense, and the industry quietly moved on.

There’s probably a similar “timeless principle” in your stack right now that’s actually a snapshot of 2015’s hardware tradeoffs. Worth looking for.

Still running Hadoop on cloud VMs because that’s how you learned it? The math has changed. S3 + Iceberg + ephemeral compute is cheaper, faster to operate, and won’t wake you up to restart a NameNode.

Your Prometheus Bill Isn't Prometheus. It's Cardinality.

Ilia Gusev — Wed, 22 Apr 2026 14:02:00 GMT

Prometheus is free. Thanos is free. Grafana is free. And yet the observability line item on your cloud bill keeps climbing. Compute for the query path, memory for the head block, S3 for long-term storage, egress charges when dashboards span clusters - the tools are open source but the infrastructure around them is very much not.

The thing nobody tells you when they hand you a Prometheus stack is that the cost driver is almost never “more metrics.” It’s cardinality. The number of unique time series - not raw data points, not storage bytes - is what determines how much RAM your Prometheus needs, how fast your queries are, and how much you pay S3 every month.

And cardinality explodes quietly. A developer adds a pod_template_hash label to track which deployment version a metric came from. Another adds request_id because they wanted to trace something. A third exports a Go runtime metric with the goroutine ID as a label. Each of these looks innocent locally. Together they can 100x your series count over a quarter.

Step 1: find the metrics that are eating you alive

Prometheus can tell you exactly which metrics are the worst offenders. Run this against your Prometheus or Thanos Querier:

topk(20, count by (__name__)({__name__!=""}))

This ranks metric names by how many time series they produce. The top 20 usually contains 80% of your total series count. Write down the names.

Then drill into the worst one. Say http_requests_total is at the top with 400,000 series. Which label is blowing it up?

topk(10, count by (pod)({__name__="http_requests_total"}))
topk(10, count by (path)({__name__="http_requests_total"}))
topk(10, count by (status_code)({__name__="http_requests_total"}))

One of these breakdowns will reveal the offender - usually a high-cardinality label that shouldn’t exist. Common culprits: a path label that contains user IDs, a user_id label that contains user IDs, a pod_template_hash that changes every deploy, a request_id that changes every request. Each of these turns a reasonable 50-series metric into a 50,000-series metric.

Histogram metrics (the ones ending in _bucket) deserve their own investigation. Each bucket is a separate series, so a histogram with 10 buckets and 100 label combinations is already 1,000 series before you add anything.

Step 2: attribute the cost to the team that created it

Once you know where the series are, the next question is who owns them. “We spent $3,200 on observability last month” is not an actionable number. “Team alpha spent $1,400, team bravo spent $900, team charlie spent $300” is.

The pattern is to require a team label on every metric. You can enforce this in two ways:

In the application. Your metrics library adds a team label at registration time, pulled from an environment variable that your deployment pipeline sets. Every metric from that service inherits the label. This is the cleanest but requires changes in every service.

At the scrape layer via relabel_configs. Your ServiceMonitor or scrape config reads a teamlabel from the pod annotations and copies it onto every metric from that pod. No code changes required.

Once you have the label, you can query cost-relevant breakdowns:

# Hot series in memory, by team
sum(prometheus_tsdb_head_series) by (team)

# Ingestion rate (samples per second), by team
sum(rate(prometheus_tsdb_head_samples_appended_total[5m])) by (team)

Make these dashboards public. The moment a team sees its own line on a “monthly observability cost per team” chart, conversations about cardinality get much easier.

Step 3: downsample aggressively with Thanos

Raw resolution data - one sample every 15 seconds - is useful for about 30 days. After that, nobody is debugging a specific incident at second-level granularity; they’re looking at trends. Thanos Compactor can aggregate your data into coarser resolutions as it ages:

args:
  - --retention.resolution-raw=30d
  - --retention.resolution-5m=1y
  - --retention.resolution-1h=5y
  - --downsampling.disable=false

This is roughly equivalent to saying “keep 30 days of raw data, then 5-minute rollups for a year, then hourly rollups for 5 years.” For storage purposes, 5-minute resolution cuts your sample count by 20x compared to 15-second raw data. Hourly resolution cuts it by another 12x on top of that.

Your long-retention S3 bill gets a lot cheaper. Your queries over long time windows also speed up dramatically - rate(errors[90d]) no longer has to scan 518,400 raw samples per series, it scans 26,000 5-minute samples.

Step 4: drop the labels you don’t query

The most underused optimization is metric_relabel_configs with action: labeldrop. This strips a label at scrape time, before anything is written to storage. Series that differ only in that label collapse into one.

metric_relabel_configs:
  - action: labeldrop
    regex: pod_template_hash

This one line, on a busy cluster, can cut your series count by a factor of 5. Every time a deployment rolls out, pod_template_hash changes, and if nothing drops it, every metric from that deployment becomes a new set of series. The old series don’t delete - they just stop receiving samples and linger in storage until they age out.

Other common drops: instance_id if you don’t need per-instance granularity, git_commit if you’re exposing it as a label (make it an info metric instead), any label whose value is a random string.

The rule I use: if a label has never been in a by() clause in any query I’ve actually run, it should be dropped at ingestion. Measure your query log for a month and you’ll be surprised at how few labels are actually used.

Making this a continuous process

Cardinality control is not a one-time cleanup. New services get added. Old services grow new metrics. Labels drift. The only way to keep it from regressing is to make cardinality visible, reviewed, and owned.

The minimum setup that actually works:

A dashboard showing total series count over time, broken down by team and metric name.
An alert that fires when a single metric crosses 100,000 series, or when total series grow by more than 10% week-over-week.
A relabel_configs allowlist or denylist that’s reviewed as part of every new service onboarding.
A monthly “cost per team” report sent to engineering leads.

None of this is hard. All of it is unglamorous. The teams that skip it end up with observability bills that look like a runaway data warehouse - and the solution is never “buy more storage,” it’s always “stop writing data nobody reads.”

If your Prometheus memory usage climbs every week and nobody knows why, start with topk(20, count by (__name__)({__name__!=”“})) - you’ll find the offender in five minutes.

Karpenter Beyond the Demo: The Patterns That Save Real Money

Ilia Gusev — Tue, 21 Apr 2026 14:03:15 GMT

Welcome back to Podo Stack. Karpenter is one of those tools that looks trivial in the intro tutorial - kubectl apply a NodePool, watch nodes appear, feel clever. Then you run it in production for a month and discover that your workloads drift into the wrong zones, your spot strategy silently falls back to full on-demand pricing, and your consolidation fights with Descheduler until half your cluster is in permanent pod churn.

This week: five patterns that separate a demo-grade Karpenter setup from one that actually saves you money in production.

Here’s what’s good this week.

🚀 The Foundation: NodePool and EC2NodeClass Separation

Stop mixing policy with plumbing

When Karpenter moved from the v1alpha5 Provisioner to v1beta1, the monolithic CRD got split into two. NodePool owns the autoscaling policy: which instance types are allowed, what capacity types (spot/on-demand), consolidation rules, resource limits. EC2NodeClass owns the AWS plumbing: AMI family, IAM role, subnet and security group selectors, user data.

This split isn’t cosmetic. It lets platform teams own the infrastructure concerns - which subnets, which IAM role, which AMI - while letting application teams define their own scheduling policies without touching security-sensitive fields.

The NodePool uses attribute-based requirements instead of hardcoded instance types. You don’t list m5.large, m5.xlarge, m5.2xlarge - you say “instance-category in [c, m, r], instance-generation > 2, arch in [amd64, arm64]”. Karpenter picks the cheapest available instance that fits.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
      nodeClassRef:
        name: default-node-class
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s
    expireAfter: 720h

expireAfter: 720h is worth highlighting. It forces every node to rotate within 30 days. You get patched AMIs, you get fresh kernels, and you stop accumulating weird node-local state - all without cron or human intervention.

💎 Hidden Gem: Spot-to-On-Demand Fallback with Weights

One knob turns spot from “dev-only” into “run everything on it”

The biggest fear teams have about Spot in production is capacity: what if AWS has no c6i.large in us-east-1a when I need it? The answer isn’t “don’t use Spot.” The answer is two NodePools and the weight field.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: spot-pool
spec:
  weight: 80    # higher = higher priority   (spot-pool)
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: ondemand-pool
spec:
  weight: 20           # fallback
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

When a pod goes Pending, Karpenter tries the lowest-weight NodePool first. If it can’t find spot capacity - no matching instance type is available right now - it falls through to the next NodePool and provisions an on-demand node. The pod never notices. Your bill notices when spot is plentiful (70% off on-demand) and quietly recovers when spot is scarce.

The subtle trick here is instance flexibility. If you lock your spot NodePool to a single instance family (c6i only), spot availability is fragile. Open it up to categories c, m, r across multiple generations and architectures, and Karpenter has dozens of pools to pick from. Spot interruption rates drop because you’re not concentrated on a single pool.

Pair this with correct PodDisruptionBudgets on your critical workloads and Karpenter will handle spot interruptions gracefully - cordon the doomed node, spin up a replacement, drain within PDB limits.

🔥 The Trap: Topology Spread Constraints with `DoNotSchedule`

topologySpreadConstraints looks like the obvious way to tell Karpenter “spread my pods across zones.” And it works - until the day it deadlocks your scaling and the on-call engineer spends three hours figuring out why perfectly healthy pods are stuck Pending.

The trap is whenUnsatisfiable: DoNotSchedule. That’s a hard constraint. Here’s the scenario: maxSkew: 1, three zones, 10 pods in A, 10 in B, 9 in C. A new pod arrives. Karpenter reads the TSC, sees the new pod must go to zone C to keep maxSkew: 1. Karpenter tries to provision a node in C - and the spot pool in C is exhausted. Karpenter cannot fall back to A or B because that would violate DoNotSchedule. The pod sits Pending forever. Your dashboards go yellow. Your PagerDuty goes off.

For most workloads, use ScheduleAnyway instead. It’s a soft hint: Karpenter tries to spread evenly, but if it can’t find capacity in the “right” zone, it provisions where it can. You sleep at night.

Reserve DoNotSchedule for genuinely quorum-sensitive systems: etcd, ZooKeeper, Kafka controllers. For those, losing zone balance is worse than a failed schedule. For your API servers? ScheduleAnyway is the right default.

One more trap: topologyKey: kubernetes.io/hostname with maxSkew: 1. Karpenter reads this literally - “at most one extra pod per node” - and provisions one node per pod. Your 10-pod deployment becomes 10 nodes running at 8% utilization. Use podAntiAffinity if you want “never colocate,” not TSC.

⚡ Deep Dive: SpotToSpotConsolidation

Karpenter becomes a proactive fleet manager instead of a reactive scaler

By default, Karpenter refuses to move a pod from one spot node to another. The logic is sensible: both nodes are interruptible, so moving between them buys no stability, and the pod churn has a real cost. Consolidation only touches empty nodes or moves pods from on-demand to spot.

Turn on SpotToSpotConsolidation and the calculus changes. Karpenter will now actively hunt for two things:

Cheaper spot pools. Spot prices fluctuate constantly. Your pod is on a m5.large at $0.05/hour. A compatible c6a.large in the same zone just dropped to $0.03/hour. Karpenter provisions the cheaper node, drains the pod within your PDB, and kills the old one. 40% savings, transparent to the app.

Lower-interruption pools. AWS exposes interruption frequency forecasts per spot pool. If your current pool is trending toward “high interruption risk” and a similarly priced pool is trending “low risk,” Karpenter rebalances you before the interruption notice arrives. You’re paying the same price for a more stable fleet.

The trade-off is controlled churn. Your pods will be rescheduled more often. This means two hard requirements: PodDisruptionBudgets on everything that can’t tolerate instant termination, and applications that are actually stateless - or idempotent at startup. If your app takes 90 seconds to warm caches on boot, SpotToSpot rotation will hurt. Measure before you enable.

💣 The One-Liner: Delete Your Descheduler

kubectl delete deployment descheduler -n kube-system

If you’re running Descheduler alongside Karpenter, stop. It’s an anti-pattern that creates constant pod churn without the payoff you expect.

Descheduler was built to compensate for Cluster Autoscaler’s passivity. CA only removes a node when it’s completely empty, so Descheduler evicts pods from low-utilization nodes to help CA drain them. That architecture made sense in 2019.

Karpenter doesn’t wait for empty nodes. Its built-in Consolidation does the same job across the whole cluster: it looks at every node, finds nodes that can be merged into fewer, larger ones, and performs the drain-and-replace atomically. The key word is atomically - Karpenter decides which pod goes where before it drains, so the reshuffling is a single coordinated operation.

Run Descheduler on top and the two tools fight. Descheduler evicts a pod to balance utilization. Kube-scheduler places it on another node. A minute later, Karpenter’s Consolidation decides the new placement is suboptimal and starts its own drain. The pod bounces three times in ten minutes and your users see 503s.

Same story for other Descheduler strategies. RemovePodsViolatingNodeTaints? Karpenter’s Drift detection handles taint changes by replacing the whole node. RemovePodsViolatingInterPodAntiAffinity? Karpenter’s Consolidation respects anti-affinity when it picks placement. Everything Descheduler does, Karpenter does better and without the churn.

If you’re on Karpenter, delete Descheduler. Your pods will thank you.

Questions? Feedback? Reply to this email. I read every one.

Podo Stack - Ripe for Prod.

Invisible Indexes: The Safest Way to Delete an Index You're Not Sure About

Ilia Gusev — Fri, 17 Apr 2026 14:02:18 GMT

Here’s the classic production database problem. You find an index that looks useless. ix_orders_user_status - nobody in the team remembers adding it. It’s 4 GB. The query that supposedly uses it runs once a day for a report nobody reads anymore.

But you don’t want to drop it. Because the one time in two years that someone runs that report, dropping the index will turn a 200ms query into a 40-minute full scan, and you’ll hear about it from the CFO.

MySQL 8.0 has a feature for this exact problem. It’s called an invisible index, and it solves the “maybe I should drop this” problem without the risk. You hide the index from the optimizer while keeping it physically on disk. If nothing breaks for a few days, you drop it. If something breaks, you flip it visible again in one ALTER TABLE.

What “invisible” actually means

An invisible index is a normal B-tree index that the optimizer pretends doesn’t exist. Every INSERT, UPDATE, and DELETE still maintains it - the writes still hit the index pages. Every query plan is generated as if the index weren’t there.

ALTER TABLE orders ALTER INDEX ix_user_status INVISIBLE;

That’s it. One statement. No data movement, no rewriting, no table rebuild. The index goes from “visible to the optimizer” to “invisible to the optimizer” instantly.

And if you need it back:

ALTER TABLE orders ALTER INDEX ix_user_status VISIBLE;

Also instant. The index was never physically gone - it was just hidden. You flip the bit, the optimizer can see it again, query plans using it start working immediately.

The three things this is perfect for

Auditing unused indexes. Over years of schema evolution, dead indexes accumulate. Some are duplicates (two indexes with the same leading columns in different order). Some were added “just in case” by someone who left. Dropping them feels risky because you can’t be sure nothing depends on them. Mark them invisible for a week, watch for regressions, then drop the ones that caused zero complaints.

Forcing the optimizer to try a different plan. Sometimes the optimizer stubbornly picks a bad index. You suspect another index would be better, but you can’t remove the “bad” one because other queries need it. Make it invisible temporarily. The optimizer is forced to pick a different plan. If the new plan is better across the board, congratulations - you have evidence for a refactor. If it’s worse, flip it back.

Rolling out a new index carefully. Create the new index as invisible. It gets maintained on all writes but doesn’t affect any query plans. Run your performance tests on a replica with real traffic. If the metrics look good, flip it visible on production. If not, drop it without ever having risked a plan regression.

The workflow

The whole process looks like this:

-- 1. Pick a candidate via SHOW INDEX and slow query log
SHOW INDEX FROM orders WHERE Key_name = 'ix_user_status';

-- 2. Make it invisible (instant)
ALTER TABLE orders ALTER INDEX ix_user_status INVISIBLE;

-- 3. Wait 24-48 hours. Watch your metrics:
--    - p95 and p99 latency per endpoint
--    - handler reads per second
--    - slow query log entries
--    - any alerts from your APM

-- 4a. If everything is fine, drop it
ALTER TABLE orders DROP INDEX ix_user_status;

-- 4b. If something is worse, flip it back
ALTER TABLE orders ALTER INDEX ix_user_status VISIBLE;

The reason this is so much better than the straight DROP INDEX approach is reversibility. DROP INDEX on a large table is not free. The rollback is CREATE INDEX, which takes minutes or hours and might lock the table depending on your MySQL version and storage engine settings. Invisible-then-drop gives you a cheap, instant rollback during the observation window, and only after you’re confident do you pay the DDL cost of actually dropping it.

The catches

Primary keys and unique indexes can’t be made invisible. They participate in constraint enforcement, not just query optimization, so MySQL refuses. You’ll get an error if you try.

Writes still hit the index. This is the big one. The index is invisible to reads but alive for writes. If your goal is to reduce write amplification on a heavy OLTP table, making the index invisible does not help. You have to actually drop it to see the write savings. Invisible is for testing the read path, not the write path.

The query hint escape hatch. If you’re desperate to test a query with the invisible index anyway, you can force it with SET SESSION optimizer_switch='use_invisible_indexes=on';. This makes invisible indexes visible for your session only. Useful for debugging “would this index help if I re-enabled it?” without affecting production traffic.

Replica considerations. Make sure you understand how your replication topology handles DDL. On a typical MySQL replication setup, the ALTER TABLE ... INVISIBLE propagates through the binary log like any other DDL. If you run multi-source or group replication, verify the behavior matches your expectations before you rely on it.

Why it matters

Before MySQL 8.0, index management was a one-way door. You either kept the index (paying its storage cost and write overhead) or dropped it (risking a plan regression you wouldn’t notice until it was too late to easily recover). Invisible indexes turn that binary decision into a reversible experiment.

The same DBA pattern works in both directions - testing removals and testing additions. Combined with a proper slow query log and per-query metrics, you can run real A/B experiments on your index set without ever risking production stability.

If you’re running MySQL 8.0 and you’ve never used this feature, the next time someone asks “can we drop this index?” the answer should be “let’s make it invisible first.”

Still dropping indexes and hoping for the best? Flip them invisible first - it’s one ALTER TABLE and it buys you a real rollback window.

AUTO_INCREMENT Is Silently Throttling Your Writes

Ilia Gusev — Wed, 15 Apr 2026 14:02:50 GMT

For years, AUTO_INCREMENT has been the boring, obvious choice. You declare id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, and MySQL hands out unique numbers. Nobody questions it because it just works.

Then your write volume crosses some invisible line - usually in the tens of thousands of inserts per second - and things get strange. Your p99 insert latency starts climbing. Your CPU looks fine. Your I/O looks fine. But inserts are queueing, and you can’t figure out why.

The culprit is the auto-increment counter itself. Under the hood, it’s a lock. And locks don’t scale.

The hidden mechanism

When you insert a row into a table with AUTO_INCREMENT, InnoDB needs to hand you a unique monotonically increasing integer. The way it does that is with a special lock called the AUTO-INC lock. The lock is acquired, the counter is incremented, the new value is stored in your row, the lock is released.

In the default innodb_autoinc_lock_mode = 1 (”consecutive”), the lock is held only for the duration of the increment - brief, but still serialized across all concurrent inserts into that table. Under heavy concurrency, you end up with threads queueing behind each other for nothing but their turn to get a number.

innodb_autoinc_lock_mode = 2 (”interleaved”) relaxes this and avoids the lock for simple inserts, but at the cost of non-consecutive IDs within a transaction. If anything in your codebase or replication tooling assumes contiguous IDs, mode 2 will break it.

Either way, the AUTO-INC counter is a single point of serialization. It’s invisible until you’re doing tens of thousands of writes per second, and then it shows up as insert latency that no amount of CPU or disk tuning can fix.

When it stops scaling

There are three scenarios where AUTO_INCREMENT reliably breaks down:

Single-table write storms. Your event log, your audit trail, your analytics event sink - anything that funnels writes from many sources into one table. The AUTO-INC lock becomes the serialization point for everything.

Multi-master replication. If you’re running MySQL group replication or multi-source replication with writes on multiple nodes, two masters can hand out the same ID at roughly the same time. The classic fix is auto_increment_increment and auto_increment_offset - master A hands out 1, 3, 5, 7; master B hands out 2, 4, 6, 8 - but that only works for exactly two masters and breaks when you add a third. And the moment you fail over and the offset assignment drifts, you get duplicate key errors in replication.

Distributed IDs as external references. The AUTO_INCREMENT ID of your users table starts appearing in URLs, in Kafka messages, in caches, in other services’ foreign keys. Now you have a problem if you ever need to split the users table across shards. There’s no way to shard cleanly by an integer that was handed out sequentially - all your recent users end up on the same shard.

The distributed alternatives

When AUTO_INCREMENT is the bottleneck, the answer is to move ID generation out of the storage layer. You have three common choices:

UUID (version 4). 128 bits of random data. Guaranteed unique across any number of writers without coordination. The downside for MySQL specifically: UUIDs are effectively random, so inserting them into the primary B-tree causes scattered writes and destroys your cache locality. This is why naive UUID primary keys kill MySQL performance on large tables. UUIDv7 (time-ordered) fixes most of this and should be the default when you need a globally unique ID in 2026.

ULID. 128 bits, but the high bits are a millisecond timestamp. The result sorts roughly in time order, which makes it much friendlier to B-tree indexes. Same no-coordination property as UUID. Slightly less standard than UUID but widely supported.

Snowflake-style IDs. 64 bits: timestamp + machine ID + sequence. The ID fits in a BIGINT, sorts by time, and can be generated by any worker as long as each worker has a unique machine ID. Twitter invented this, Discord uses it, Instagram uses a variant. It’s the choice when you want distributed generation but also want numeric IDs that fit in existing schemas.

For any new schema at scale, I’d default to ULID or UUIDv7 for the primary key and never look back. The storage overhead - 16 bytes versus 8 for BIGINT - is a rounding error compared to the scaling properties you get.

The migration trap

Here’s the part people underestimate. Switching from AUTO_INCREMENT to distributed IDs on an existing schema is a whole project, not a one-line change.

Your foreign keys all point at integer IDs. Your indexes are sized for BIGINT. Your application code parses IDs as integers. Your event payloads serialize them as numbers. Every one of those assumptions needs to be revisited. The safe pattern is to add a new column - public_id BINARY(16) for UUID, or VARCHAR(26) for ULID - populate it for new rows first, backfill in the background, switch read paths one at a time, and only then retire the integer ID as the external reference.

The old integer ID can stay as a clustered primary key for InnoDB storage efficiency. You use the new distributed ID as the external-facing reference. This hybrid is often the right answer - you get the InnoDB benefit of a sequential integer clustered key, plus the distributed ID for everything that crosses a service boundary.

A debugging checklist

Before you conclude that AUTO_INCREMENT is your bottleneck, verify. The symptoms overlap with other issues.

Run SHOW ENGINE INNODB STATUS\G and look for threads waiting on the AUTO-INC lock.
Query performance_schema.data_locks filtered to your hot table.
Check innodb_autoinc_lock_mode - if it’s 1 and you’re seeing contention, try mode 2 after confirming nothing depends on consecutive IDs within a transaction.
Measure the actual benefit. If switching to distributed IDs adds 2ms to every insert for network round-trips to an ID service but removes a 20ms lock wait, you’re ahead. If it adds 5ms and removes 3ms, you’re worse off. Measure before you migrate.

AUTO_INCREMENT is fine for most workloads. It’s the hidden lock in the corner for the few that outgrow it.

Found this useful? Subscribe to Podo Stack for weekly database patterns, Kubernetes internals, and Cloud Native tools ripe for production.

Your Business Logic Is Spread Across Cron, Queues, and State Flags. Temporal Kills All Three.

Ilia Gusev — Tue, 14 Apr 2026 14:03:17 GMT

Here’s a workflow you’ve probably written before. Register a user. Send a welcome email.

Wait 24 hours. If they didn’t finish onboarding, send a reminder. Wait 3 more days. If still nothing, send a final nudge. Otherwise give them a bonus.

Now implement that. Where does the “wait 24 hours” live? In a cron job that polls a database every minute? In a delayed message queue that might drop the job during a broker restart? In a state flag column plus a scheduler plus a retry table plus a dead-letter queue?

This is what teams call the state management hell. The business logic - six lines of pseudocode in your head - gets smeared across five systems. Debugging means grepping across all of them. Changing “wait 3 days” to “wait 4 days” means a migration script for in-flight state. Testing it means mocking time and half your infrastructure.

Temporal solves this by making durable workflows a primitive. You write the six lines as six lines. The platform handles everything else.

The Pattern: Sequential Code for Distributed Logic

Your workflow is just code. Temporal makes it durable.

A Temporal workflow looks like an ordinary function. It calls activities, waits on timers, reacts to signals. The magic is that the function can be interrupted at any point - server crash, deployment, network blip - and resume exactly where it left off. No state flags. No cron. No “which step was I on again?”

func OnboardingWorkflow(ctx workflow.Context, userID string) error {
    // Step 1-2: activities for side effects
    if err := workflow.ExecuteActivity(ctx, CreateUser, userID).Get(ctx, nil); err != nil {
        return err
    }
    workflow.ExecuteActivity(ctx, SendWelcomeEmail, userID).Get(ctx, nil)

    // Step 3: literally sleep for 24 hours
    workflow.Sleep(ctx, 24*time.Hour)

    // Step 4: remind them
    workflow.ExecuteActivity(ctx, SendProfileReminder, userID).Get(ctx, nil)

    // Step 5: wait for a signal or timeout after 3 days
    profileCh := workflow.GetSignalChannel(ctx, "profile_filled")
    selector := workflow.NewSelector(ctx)
    filled := false
    selector.AddReceive(profileCh, func(c workflow.ReceiveChannel, more bool) { filled = true })
    selector.AddFuture(workflow.NewTimer(ctx, 72*time.Hour), func(f workflow.Future) {})
    selector.Select(ctx)

    if filled {
        return workflow.ExecuteActivity(ctx, GiveBonus, userID).Get(ctx, nil)
    }
    return workflow.ExecuteActivity(ctx, SendFinalNudge, userID).Get(ctx, nil)
}

That workflow.Sleep(ctx, 24*time.Hour) is not sleeping your process. It’s a durable timer stored by the Temporal server. The worker can crash, the pod can be rescheduled, the region can fail over - the timer fires on schedule because it lives outside your process.

The trade-off is real. Temporal adds infrastructure: a server cluster (or Temporal Cloud), a persistence backend (Cassandra, PostgreSQL, or MySQL), and workers you run yourself. It’s not free. But if you’re already running cron plus delayed queues plus state machines, you were paying for durability anyway - just in duplicated code and operational pain.

Hidden Gem: Determinism and Replay

The worker doesn’t “resume” - it re-executes and skips

Here’s the part that catches everyone off guard. When a worker picks up a workflow after a crash, it doesn’t load state from a snapshot. It re-executes your workflow function from the top. Every call to ExecuteActivity, every Sleep, every Select - runs again.

Why isn’t this disastrous? Because Temporal records every completed activity result in an event history. When the workflow re-runs, each activity call is intercepted: if the event history already has a result, Temporal returns it immediately without calling the activity again. The function walks past the already-completed steps at CPU speed until it hits the last unfinished point.

This is why workflow code must be deterministic. Same input, same path, every time. No rand.Float64(), no time.Now(), no direct HTTP calls, no iteration over Go maps without sorting. All non-determinism has to live in activities, where it’s recorded once and replayed from history.

WorkflowStarted                       (input: userID=alice)
ActivityTaskScheduled                 (CreateUser)
ActivityTaskCompleted                 (CreateUser → ok)
ActivityTaskScheduled                 (SendWelcomeEmail)
ActivityTaskCompleted                 (SendWelcomeEmail → ok)
TimerStarted                          (24h)
TimerFired                            ← worker crashed here
ActivityTaskScheduled                 (SendProfileReminder)
...

Re-execution is free. Determinism is the contract that makes it work.

The Showdown: Workflow vs Activity

Workflow: Deterministic. Orchestration logic only. Cannot make network calls, read files, or get the current time directly. Re-executes from event history on recovery. Lives as long as your business process - minutes, days, years.

Activity: Non-deterministic. Does the actual work - database queries, API calls, file uploads. Runs at-least-once (your activity must be idempotent or use Temporal’s idempotency keys). Has its own retry policy, timeout, and heartbeat. Lives only as long as one execution.

The mental model: workflows describe what happens and in what order. Activities do the actual thing. If you find yourself wanting to call a REST API inside a workflow, that’s a smell - wrap it in an activity. If you find yourself tracking “which step am I on” inside an activity, you’ve reinvented the workflow in the wrong place.

Activities get the observability for free. Every invocation is a row in the event history: scheduled, started, completed or failed, with payload and timestamp. You can see exactly which call timed out, which one retried, how long each took. No custom logging infrastructure needed.

Deep Dive: Retry Policies, Timeouts, and Heartbeats

Three different timeouts. Most people only know one.

When you call an activity, Temporal gives you three separate timeout knobs, and getting them wrong is the number one source of confusion.

Start-to-Close is the one people actually mean when they say "timeout." How long can a single attempt take? Set it to your p99 plus headroom. If the activity exceeds this, Temporal treats that attempt as failed and applies the retry policy.

Schedule-to-Close is the end-to-end deadline across all retries. If you set Start-to-Close to 30 seconds and the retry policy allows up to 10 attempts, Schedule-to-Close puts a hard ceiling on the whole thing - say, 10 minutes. After that, no matter how many retries remain, the activity gives up.

Schedule-to-Start is how long the task can sit in the task queue before a worker picks it up. This is the alarm bell for worker fleet problems: if this fires, your workers are overloaded or dead, not your business logic.

The retry policy itself is surprisingly rich. Initial interval, backoff coefficient, maximum interval, maximum attempts - and a list of non-retryable error types so you don’t waste retries on ValidationError. For long-running activities (ETL jobs, file uploads, ML training), you use heartbeats: the activity calls RecordHeartbeat periodically, and if Temporal doesn’t see one for HeartbeatTimeout, it assumes the worker is dead and reschedules.

retryPolicy:
  initialInterval: 1s
  backoffCoefficient: 2.0
  maximumInterval: 100s
  maximumAttempts: 10
  nonRetryableErrorTypes:
    - ValidationError
    - PermissionDenied

The One-Liner: Versioning Workflows in Production

v := workflow.GetVersion(ctx, "add-bonus-check", workflow.DefaultVersion, 1)

This is the single most important line if you ever plan to change a workflow while real workflows are in flight. And you will.

Remember the determinism rule - workflows re-execute from history. If you deploy new code that adds a step between step 2 and step 3, any workflow that was mid-flight on the old code will replay with the new code and hit a non-deterministic execution error. The history says “after step 2 we started timer X” and the new code says “after step 2 we called activity Y”. Temporal refuses to continue and the workflow is stuck.

GetVersion is how you branch safely. On first execution, it records the version in history. On replay, it returns the recorded version. You use it to fork the code path: if v == DefaultVersion { oldPath() } else { newPath() }. Old workflows keep running the old path; new ones get the new path. Once all old workflows have drained, you clean up the GetVersion calls.

Alternatives: use workflow reset to replay from a specific event, or just launch a new workflow type and let the old one finish. But in practice GetVersion is the pragmatic default for any non-trivial change to a running workflow.

Denormalization: When JOINs Kill Your Read Performance

Ilia Gusev — Fri, 10 Apr 2026 09:01:43 GMT

Every database course teaches normalization first. Third normal form. No redundancy. Single source of truth. And it’s the right starting point - until your dashboards take 12 seconds to load because the query joins seven tables to show a username next to an order.

Normalization optimizes for writes. Denormalization optimizes for reads. Most production systems are 90%+ reads. You do the math.

Subscribe now

I’m not saying throw away your foreign keys. I’m saying there are specific, measurable situations where copying a column into another table saves you seconds of query time and megabytes of memory. And that trade-off is worth making consciously.

The actual cost of JOINs at scale

A single JOIN between two indexed tables is fast. Two JOINs - still fine. But production queries rarely stop there.

Consider an order details page: orders JOIN users JOIN products JOIN shipping_addresses JOIN payment_methods JOIN order_items. Six tables. Each JOIN multiplies the work - fetching pages from different parts of disk, building hash tables in memory, matching keys across indexes.

At 1,000 requests per second, those JOINs don’t just cost time. They cost buffer pool pages, CPU cycles, and connection slots. The query that takes 15ms at low traffic takes 200ms under load because every JOIN competes for the same memory.

The pattern: embed what you read together

The core idea is simple. If you always read user_name when you read orders - put user_name in the orders table.

ALTER TABLE orders
  ADD COLUMN customer_name VARCHAR(100),
  ADD COLUMN customer_country CHAR(2);

Now your order list query goes from:

SELECT o.id, o.total, u.name, u.country
FROM orders o
JOIN users u ON o.user_id = u.id
WHERE o.created_at > '2026-01-01';

To:

SELECT id, total, customer_name, customer_country
FROM orders
WHERE created_at > '2026-01-01';

One table. One index scan. No JOIN. The query that used to touch two B-trees now touches one.

When denormalization makes sense

Not every JOIN deserves elimination. Denormalize when all three conditions are true:

Read-heavy ratio. The table is read 10-100x more than it’s written. E-commerce order history, analytics dashboards, reporting tables, user activity feeds. If writes are frequent, the sync cost eats your read gains.

The JOIN is in the hot path. Not every slow query matters. Denormalize the queries that users wait on - page loads, API responses, search results. If it’s a nightly batch job, the JOIN is fine.

The source data changes rarely. Customer names change infrequently. Countries almost never change. Product titles change sometimes. Prices change often. Denormalize the stable columns. Leave the volatile ones normalized.

Keeping the data in sync

This is where denormalization gets its bad reputation. You’ve duplicated data. Now you have two sources of truth. If the user changes their name, the users table updates but orders.customer_name doesn’t - unless you handle it.

Three patterns, from simplest to most reliable:

Triggers. The database itself keeps the copy in sync. Fast, automatic, invisible to your application:

CREATE TRIGGER sync_customer_name
AFTER UPDATE ON users
FOR EACH ROW
  UPDATE orders
  SET customer_name = NEW.name
  WHERE user_id = NEW.id;

The downside: triggers are hard to debug and easy to forget about. They don’t show up in your application code.

Application-level sync. Your service updates both tables in the same transaction. Explicit, visible in code, reviewable in PRs. But you have to remember to do it everywhere.

Background reconciliation. A scheduled job compares the denormalized columns with their source tables and fixes drift. This is your safety net, not your primary sync:

UPDATE orders o
JOIN users u ON o.user_id = u.id
SET o.customer_name = u.name
WHERE o.customer_name != u.name;

Run it hourly or nightly. It catches whatever the triggers or application code missed.

In practice, most teams use triggers for the primary sync and a background job as a safety net. It’s belt and suspenders, but stale data in production is worse than a few extra writes.

Measuring the impact

Don’t denormalize on gut feeling. Measure before and after:

-- Before: JOIN query
EXPLAIN ANALYZE
SELECT o.id, o.total, u.name
FROM orders o JOIN users u ON o.user_id = u.id
WHERE o.created_at > '2026-01-01';

-- After: denormalized query
EXPLAIN ANALYZE
SELECT id, total, customer_name
FROM orders
WHERE created_at > '2026-01-01';

You should see the number of rows examined drop, the execution time shrink, and the query plan simplify from a nested loop or hash join to a simple range scan.

Also check buffer pool pressure - fewer pages loaded per query means more room for other indexes and queries.

When not to denormalize

Highly volatile source data. If the source column changes multiple times per hour per row, the sync cost dominates. You’re writing more to keep the copy current than you’re saving on reads.

Low traffic. If the query runs 50 times a day, the JOIN overhead is irrelevant. Don’t add complexity for a problem that doesn’t exist yet.

Analytics you can pre-compute. If you need total_orders per user, a materialized view or a summary table is cleaner than a counter column that you increment on every insert.

When vertical partitioning solves it. Sometimes the real problem isn’t JOINs - it’s that your table is too wide and queries load columns they don’t need. Split the table first, denormalize second.

The mindset shift

Normalization is a design principle. Denormalization is an engineering decision. You normalize when modeling the domain. You denormalize when serving the users.

The best schemas I’ve seen do both: normalized source tables that are the system of record, with denormalized read tables or columns that serve the hot path. The source stays clean. The reads stay fast.

It’s not about breaking the rules. It’s about knowing which rules exist for the textbook and which exist for production.

Found this useful? Subscribe to Podo Stack for weekly database patterns, Kubernetes internals, and Cloud Native tools ripe for production.

Subscribe now

Running a dashboard that joins seven tables on every page load? Pick the one JOIN that hurts most, duplicate one column, and measure the difference. Start small.

Prefix Indexes: Stop Indexing 255 Characters When 16 Is Enough

Ilia Gusev — Wed, 08 Apr 2026 09:01:21 GMT

Your VARCHAR(255) column has an index. That index stores every byte of every value. For 10 million rows of email addresses, that’s roughly 2.5GB of index data.

But here’s the thing - most email lookups match on the first 12-16 characters. The rest of the string is dead weight in the index. You’re paying storage and memory for bytes the query optimizer never needs.

Subscribe now

Prefix indexes fix this. Instead of indexing the full string, you index only the first N characters. The index shrinks by 5-10x. Queries stay fast. And your InnoDB buffer pool stops wasting RAM on index pages that don’t earn their keep.

The syntax

It’s one keyword away from a regular index:

CREATE INDEX idx_email_prefix ON users (email(16));

That (16) tells MySQL: store only the first 16 characters. For a VARCHAR(255) column, that’s roughly 6% of the maximum length. The index is dramatically smaller.

You can do this on CREATE TABLE or as an ALTER TABLE. Works on VARCHAR, CHAR, TEXT, BLOB - any string type.

Choosing the right prefix length

Too short and the index isn’t selective enough - too many rows share the same prefix, so MySQL still scans thousands of entries. Too long and you lose the size benefit.

The trick is measuring selectivity. You want the prefix that captures most of the uniqueness:

-- Full column selectivity (baseline)
SELECT COUNT(DISTINCT email) / COUNT(*) AS full_selectivity
FROM users;

-- Test different prefix lengths
SELECT
  COUNT(DISTINCT LEFT(email, 8))  / COUNT(*) AS sel_8,
  COUNT(DISTINCT LEFT(email, 12)) / COUNT(*) AS sel_12,
  COUNT(DISTINCT LEFT(email, 16)) / COUNT(*) AS sel_16,
  COUNT(DISTINCT LEFT(email, 20)) / COUNT(*) AS sel_20
FROM users;

If full selectivity is 0.98 and LEFT(email, 16) gives you 0.95 - that’s close enough. You’re keeping 97% of the uniqueness at 6% of the size.

A rule of thumb: aim for 90-95% of full-column selectivity. Beyond that, you’re paying more storage for diminishing returns.

Where prefix indexes shine

The sweet spot is long strings with high uniqueness at the front:

Email addresses - user@ is usually unique enough in the first 16 chars
URLs - https://example.com/path diverges early
UUIDs stored as VARCHAR - the first 8 characters of a UUID v4 give strong selectivity
File paths - /data/region/tenant/... branches early in the tree
JSON string fields - when you’re filtering on a text column that holds serialized data

For columns like country_code (2 chars) or status (5-10 chars), prefix indexes don’t make sense - just index the full column.

Composite indexes with prefixes

You can mix prefix and full columns in a composite index:

CREATE INDEX idx_type_email ON users (account_type, email(16));

MySQL uses account_type fully, then narrows within each type using the email prefix. This is powerful for queries like WHERE account_type = 'business' AND email LIKE 'john%'.

The trade-offs

No ORDER BY optimization. MySQL can’t use a prefix index for sorting. ORDER BY email won’t benefit from email(16) - the optimizer doesn’t know the full order. If you need sorted results, you need the full-column index.

No covering index. A prefix index can’t “cover” a query because it doesn’t store the complete value. If your query is SELECT email FROM users WHERE email LIKE 'john%', MySQL still has to look up the full row to return the complete email. Pair this with a covering index if you need both.

No exact UNIQUE constraint. A prefix index on email(16) can’t enforce uniqueness on the full email - two different emails might share the same 16-character prefix. If you need uniqueness, keep a full UNIQUE index alongside the prefix index for queries.

UTF-8 prefix length is in characters, not bytes. In utf8mb4, one character can take up to 4 bytes. email(16) means 16 characters, which could be up to 64 bytes. This matters for index size calculations if your data has multi-byte characters.

Prefix indexes in PostgreSQL

PostgreSQL doesn’t support prefix indexes directly - there’s no CREATE INDEX ... (column(N)) syntax. But you get the same result with a functional index:

CREATE INDEX idx_email_prefix ON users (LEFT(email, 16));

Your queries need to use the same expression: WHERE LEFT(email, 16) = LEFT('john@example.com', 16). It’s less ergonomic than MySQL’s approach, but the performance benefit is identical.

Honestly, for PostgreSQL, partial indexes often solve the same underlying problem - a large index where most entries aren’t useful - through a different mechanism.

Quick validation

After creating a prefix index, always check two things:

-- 1. Verify the optimizer uses it
EXPLAIN SELECT * FROM users WHERE email = 'john@example.com';

-- 2. Check index size vs full index
SELECT
  index_name,
  ROUND(stat_value * @@innodb_page_size / 1024 / 1024, 2) AS size_mb
FROM mysql.innodb_index_stats
WHERE table_name = 'users'
  AND stat_name = 'size';

If EXPLAIN shows your prefix index and the size is 5-10x smaller than the full-column index - you’re done. If EXPLAIN ignores the prefix index, your prefix might be too short. Bump it up by 4 characters and test again.

Found this useful? Subscribe to Podo Stack for weekly database patterns, Kubernetes internals, and Cloud Native tools ripe for production.

Subscribe now

Still indexing full VARCHAR(255) columns when only the first 16 characters matter? One ALTER TABLE is all it takes.

Tools From the Future

Ilia Gusev — Tue, 07 Apr 2026 14:04:04 GMT

Welcome back to Podo Stack. This week: five tools that feel like they’re from the future. A runtime that abstracts away infrastructure. A GitOps promotion engine by the Argo CD team. WebAssembly challenging containers. A scheduler that squeezes real money out of idle nodes. And a CNCF standard that makes feature flags vendor-neutral.

Subscribe now

Here’s what’s good this week.

🚀 Sandbox Watch: Dapr - The Standard Library for Microservices

The problem

You’re building microservices. Every service needs state management, pub/sub, service discovery, secrets, and retries. So you import SDKs - one for Redis, one for Kafka, one for Vault, one for your cloud provider’s queue. Suddenly, your business logic is buried under infrastructure glue code. And switching from RabbitMQ to Kafka? That’s a rewrite.

The solution

Dapr (Distributed Application Runtime) runs as a sidecar next to your app and exposes all those patterns through simple HTTP/gRPC APIs. Need to save state? POST /v1.0/state/mystore. Publish an event? POST /v1.0/publish/mytopic. Invoke another service? POST /v1.0/invoke/service-b/method/checkout.

Your code doesn’t know (or care) whether the state store is Redis, PostgreSQL, or CosmosDB. That’s a YAML config swap, not a code change.

Started at Microsoft, now a CNCF Incubating project. The building blocks cover the greatest hits of distributed systems: service invocation with mTLS, state management, pub/sub, input/output bindings, actors, and distributed lock.

What it’s NOT

Dapr isn’t a service mesh. Istio and Linkerd handle L7 traffic routing and network policies. Dapr handles application-level concerns - state, events, secrets. They’re complementary. You can run both. In fact, if you’re already on Cilium for networking, Dapr slots right in on top.

The catch

The sidecar adds latency - a few milliseconds per call. For most microservices, that’s nothing. For ultra-low-latency trading systems, it’s a dealbreaker. Also, debugging gets trickier when every call goes through a sidecar. Good observability (Dapr exports OpenTelemetry traces by default) helps, but it’s still another moving part.

💎 Hidden Gem: Kargo - GitOps Finally Gets Promotion Right

GitOps has a dirty secret: nobody agrees on how to promote changes between environments. You’ve got Argo CD syncing your clusters from Git. Beautiful. But who updates that Git repo when a new image passes staging tests? A janky CI script, that’s who.

Enter Kargo

Built by the creators of Argo CD (yes, the same team), Kargo is purpose-built for continuous promotion. It introduces four concepts that make the whole thing declarative:

Warehouse - watches for new artifacts (Docker tags, Helm chart versions, Git commits)
Freight - a specific bundle of artifacts, like myapp:v1.2.3 + chart:0.5.0
Stage - represents an environment (dev, staging, prod)
Promotion - the act of moving Freight from one Stage to the next

The flow: Warehouse detects a new image tag. Creates Freight. Auto-promotes to dev. Dev passes verification. Kargo shows a “Promote to Staging” button in its UI. QA clicks it. Kargo updates the Git repo. Argo CD syncs. Done.

No shell scripts. No sed -i in CI pipelines. No “who changed that image tag in the values.yaml?”

Why it matters

If you adopted Crossplane for infrastructure-as-code and Argo CD for deployment, Kargo closes the last gap. It’s the missing piece between “CI built a new image” and “that image is running in production.”

⚔️ The Showdown: WasmEdge vs Containers

Solomon Hykes (Docker’s co-founder) tweeted it back in 2019: “If WASM+WASI existed in 2008, we wouldn’t have needed Docker.” Bold claim. Let’s see where we are.

WasmEdge is a CNCF Incubating runtime for server-side WebAssembly. It compiles Wasm modules ahead-of-time to near-native speed. And the numbers are hard to ignore.

Containers:

Cold start: seconds
Image size: 100s of MB
Isolation: Linux namespaces + cgroups
Language support: anything that runs on Linux
Ecosystem: massive, battle-tested

WasmEdge:

Cold start: milliseconds
Module size: single-digit MBs
Isolation: Wasm sandbox (no syscalls by default)
Language support: Rust, C/C++, Go (TinyGo), JS, Python (growing)
Ecosystem: early but expanding fast

The verdict

It’s not either/or. Wasm shines for edge computing, serverless functions, and plugin systems where you need thousands of isolated instances with instant startup. Containers win for complex apps with deep OS dependencies, mature debugging tools, and the massive Docker Hub ecosystem.

The practical move today: use containers for your main workloads. Explore WasmEdge for FaaS, edge processing, and SaaS plugin sandboxing. Kubernetes already supports both through OCI-compatible runtimes like crun.

🔬 The Pattern: Koordinator - Make Idle CPUs Pay Rent

The problem

Your Kubernetes nodes are 15-20% utilized. You’re paying for 100% of them. The scheduler sees requested resources, not actual usage. So those 80% “reserved but idle” CPUs just sit there, burning money.

The fix

Koordinator, from Alibaba and now a CNCF project, introduces colocation - running best-effort (BE) batch jobs on the resources your latency-sensitive (LS) services reserved but aren’t using.

The key: when the LS service actually needs those resources back, Koordinator’s node agent (Koordlet) instantly throttles or evicts the BE workloads. No performance degradation for your production services. It monitors real-time metrics - CPU cycles per instruction, cache contention, memory bandwidth - and reacts in milliseconds.

The isolation goes deeper than standard cgroups. Koordinator manages LLC (Last Level Cache) allocation and memory bandwidth to prevent “noisy neighbor” problems at the hardware level.

Real numbers

Alibaba reports going from ~15% to 50%+ cluster utilization in production. That’s not a rounding error. On a 1000-node cluster, that’s potentially hundreds of nodes you don’t need to buy.

The catch

This isn’t plug-and-play. You need a solid understanding of Linux kernel scheduling, NUMA topology, and your workload patterns. It’s a power tool for platform teams that have already outgrown basic Kubernetes resource management.

🛠️ One-Liner: OpenFeature - Feature Flags Without Vendor Lock-in

Feature flags are everywhere. But every provider (LaunchDarkly, Flagsmith, Split, PostHog) has its own SDK. Switch providers, rewrite your integration code. Sound familiar?

OpenFeature is a CNCF standard that does for feature flags what OpenTelemetry did for observability: one API, swap the backend anytime.

client := openfeature.NewClient("my-app")
enabled, _ := client.BooleanValue(ctx, "new-checkout", false, evalCtx)

That code works with LaunchDarkly today and Flagsmith tomorrow. You change one line - the provider initialization - not the hundreds of flag checks scattered through your codebase. SDKs exist for Go, Java, Python, TypeScript, .NET, PHP, and more.

🔥 The Hot Take: Cloud-Native Is Accelerating, Not Stabilizing

Look at this week’s tools. Dapr abstracts distributed patterns into HTTP calls. Kargo makes GitOps promotion declarative. WasmEdge challenges the container model itself. Koordinator squeezes 3x more value from existing hardware. OpenFeature standardizes yet another fragmented space.

Five years ago, we were still arguing about whether Kubernetes was production-ready. Now we’re building abstraction layers on top of abstraction layers - and they’re actually good. The CNCF landscape isn’t just growing, it’s maturing into composable building blocks.

My take: the teams that win in 2026 aren’t the ones running the most tools. They’re the ones picking the right abstractions early. One good choice (like adopting OpenTelemetry in 2022) saves years of migration pain later.

What’s your bet? Reply and tell me which tool here you’d adopt first.

Subscribe now

Questions? Feedback? Reply to this email. I read every one.

Redis INT vs RAW Encoding: Why Your Counters Use 10x Less Memory

Ilia Gusev — Fri, 03 Apr 2026 09:01:29 GMT

Run this in your Redis CLI:

SET counter 42
OBJECT ENCODING counter

You’ll get "int". Now do this:

SET counter "hello"
OBJECT ENCODING counter

That returns "embstr". And if you set a string longer than 44 bytes, you’ll get "raw".

Subscribe now

Same command, same key, three different internal representations. Redis picks the most efficient encoding based on what you’re actually storing. And if you don’t understand how this works, you might be using 5-10x more memory than you need to.

The three encodings

Every Redis string value uses one of three internal encodings:

INT - for values that fit in a 64-bit signed integer (-9223372036854775808 to 9223372036854775807). Redis doesn’t store the string “42” - it stores the actual number 42 as an 8-byte integer. No string overhead, no length tracking, no null terminator. Just the number.

EMBSTR - for strings up to 44 bytes. Redis allocates a single contiguous memory block for both the object header and the string data. One malloc() call, one cache line. Fast to allocate, fast to access.

RAW - for strings longer than 44 bytes. Redis uses SDS (Simple Dynamic String) - a custom string implementation that tracks length, capacity, and supports binary data. The object and SDS buffer are allocated separately - two malloc() calls, two memory regions.

Why 44 bytes?

That’s not arbitrary. A Redis object header takes 16 bytes. An SDS header for small strings takes 3 bytes. A null terminator takes 1 byte. A jemalloc allocation bucket is 64 bytes. So: 64 - 16 - 3 - 1 = 44. That’s the maximum string length that fits in a single 64-byte allocation alongside the object metadata.

At 45 bytes, Redis needs a second allocation. Two allocations mean more memory overhead, more fragmentation, and an extra pointer dereference on every access.

The shared integer pool

Here’s where it gets interesting. For integers from 0 to 9999, Redis doesn’t even allocate new objects. It maintains a pre-allocated pool of 10,000 integer objects at startup. Every key that stores the value 42 points to the same object in memory.

SET user:1:login_count 42
SET user:2:login_count 42
SET metrics:errors 42

All three keys reference the exact same Redis object. Zero additional memory per key (beyond the key entry itself). This is why counters and small numeric IDs are absurdly cheap in Redis.

You can verify this with DEBUG OBJECT:

DEBUG OBJECT user:1:login_count

Look for refcount in the output. If it’s greater than 1, you’re sharing an object from the pool. For integers 0-9999, the refcount will typically be very high because Redis itself uses these objects internally too.

The encoding switch trap

This is the part that bites people. Redis automatically converts between encodings - and it’s a one-way street for certain operations.

SET counter 100
OBJECT ENCODING counter     -- "int"

APPEND counter " requests"
OBJECT ENCODING counter     -- "raw"

-- counter is now "100 requests" and there's no going back

Once Redis converts from INT to RAW, it stays RAW even if you later set it back to a pure number. Well, actually - a fresh SET will re-evaluate and pick INT again. But APPEND, SETRANGE, and other mutation operations force a permanent encoding change.

The same thing happens with EMBSTR. Any write operation on an EMBSTR string promotes it to RAW, because EMBSTR is immutable by design - Redis can’t resize a single-allocation block without potentially moving it.

SET name "alice"
OBJECT ENCODING name        -- "embstr"

APPEND name " smith"
OBJECT ENCODING name        -- "raw"

The value is only 11 bytes, well under the 44-byte limit. But APPEND forced the conversion. If you’d done SET name "alice smith" instead, it would’ve been EMBSTR.

Practical memory optimization

Use INCR/DECR for counters, not SET. INCR mykey initializes to 0 and increments atomically. The value stays INT-encoded. Don’t do SET mykey "1" followed by string manipulation - let Redis treat it as a number.

Keep numeric IDs as numbers. SET session:abc user_id 12345 keeps INT encoding. SET session:abc user_id "user_12345" forces RAW. Over millions of keys, that adds up fast.

Avoid mutating short strings. If you’re building a string incrementally with APPEND, consider building it in your application and doing a single SET. That way Redis picks the optimal encoding based on the final value, instead of promoting through encodings as you append.

Audit your encodings. Run OBJECT ENCODING on a sample of your keys. If counters show up as RAW instead of INT, something in your code is contaminating them with string operations.

How much does it actually save?

Ballpark for 1 million keys:

INT (0-9999, shared pool): practically zero extra memory beyond key entries
INT (outside pool): ~16 bytes/value = ~16 MB
EMBSTR (20-byte strings): ~64 bytes/value = ~64 MB
RAW (100-byte strings): ~160+ bytes/value = ~160 MB

The jump from INT to RAW for a simple counter is roughly 10x. Across millions of keys, that’s gigabytes of waste.

The rule of thumb

If it’s a number, keep it a number. If it’s a short string, write it once (don’t append). If it’s a long string, there’s nothing to optimize - RAW is the only option, and that’s fine. The wins come from not accidentally promoting cheap encodings into expensive ones.

Subscribe now

Found this useful? Subscribe to Podo Stack for weekly database internals, Kubernetes patterns, and Cloud Native tools ripe for production.

Curious what encoding your Redis keys are actually using? Run OBJECT ENCODING on a few of your busiest keys - you might be surprised.

Partial Indexes in MySQL: The Trick PostgreSQL Doesn't Want You to Know

Ilia Gusev — Wed, 01 Apr 2026 09:01:46 GMT

PostgreSQL has this nice trick - CREATE INDEX ... WHERE status = 'pending'. The index only stores rows that match the condition. If you have 50 million orders and only 200,000 are pending, your index is tiny. Scans are fast. Disk usage is minimal.

Subscribe now

MySQL doesn’t have that syntax. There’s no WHERE clause in CREATE INDEX. And if you google “partial index MySQL,” you’ll mostly find people saying “just switch to PostgreSQL.”

But here’s the thing - since MySQL 8.0, you can get almost the same result. It takes a different path, but the destination is the same: a small index that only contains the rows you care about.

The core idea

Instead of telling the index which rows to include, you tell it what value to store. For rows you don’t want - you store NULL.

MySQL indexes do include NULL entries (unlike PostgreSQL’s partial indexes that skip them entirely), so you won’t get identical disk savings. But the optimizer still benefits massively because the non-NULL keys are clustered together, and your queries only scan that small portion.

The trick works through two mechanisms: functional key parts and generated stored columns.

Approach 1: functional index

Say you have an orders table with 50M rows. Only 0.4% are pending. You want to quickly find pending orders by their ID.

CREATE INDEX idx_pending
    ON orders ((IF(status = 'pending', order_id, NULL)));

The IF() expression returns the order_id for pending rows and NULL for everything else. The index stores all 50M entries, but only 200K have meaningful (non-NULL) values.

The catch? Your query has to use the exact same expression:

SELECT *
FROM orders
WHERE IF(status = 'pending', order_id, NULL) = 12345;

That’s ugly. It works, but it leaks implementation details into your application code. Your ORM won’t like it. Your teammates won’t like it either.

Approach 2: generated column (the better way)

This is what I’d actually recommend for production:

ALTER TABLE orders
  ADD COLUMN pending_id BIGINT
    AS (IF(status = 'pending', order_id, NULL)) STORED,
  ADD INDEX idx_pending_id (pending_id);

Now you have a real column - pending_id - that’s automatically maintained by MySQL. It’s order_id for pending rows, NULL for everything else. Your queries look normal:

SELECT *
FROM orders
WHERE pending_id IS NOT NULL;

-- Or find a specific pending order:
SELECT *
FROM orders
WHERE pending_id = 12345;

The optimizer picks up idx_pending_id automatically. No query hints needed. No ugly IF() in your WHERE clause. Run EXPLAIN and you’ll see the index scan hitting only the pending rows.

Approach 3: boolean flag + composite

When you need to fetch all pending orders sorted by date:

ALTER TABLE orders
  ADD COLUMN is_pending TINYINT(1)
    AS (status = 'pending') STORED,
  ADD INDEX idx_pending_date (is_pending, created_at);

Now SELECT * FROM orders WHERE is_pending = 1 ORDER BY created_at DESC uses an index range scan. The index is compact because most rows have is_pending = 0, and the optimizer skips those efficiently.

The gotchas

NULLs are still indexed. This is the biggest difference from PostgreSQL. A true partial index in Postgres with WHERE status = 'pending' physically excludes non-matching rows. MySQL’s approach stores NULL keys, so the index is larger. For a 50M row table with 200K pending rows, PostgreSQL’s partial index stores 200K entries. MySQL’s stores 50M entries (most of them NULL). Still way faster than a full index on status, but not as compact.

Exact expression match matters for functional indexes. If your index uses IF(status = 'pending', order_id, NULL) but your query uses WHERE status = 'pending', the optimizer won’t connect the dots. That’s why the generated column approach is better - it gives you a clean column name to query against.

STORED, not VIRTUAL. Generated columns can be VIRTUAL (computed on read) or STORED (physically saved). Only STORED columns can be indexed. The STORED keyword means MySQL maintains the value on every INSERT and UPDATE, so there’s a small write overhead.

Run ANALYZE TABLE after. The optimizer needs fresh statistics to know about your new index. Always ANALYZE TABLE orders; after adding the index.

When this pays off

The sweet spot is clear: a column with low cardinality where one value is rare and you query for it frequently. Some classic examples:

status = 'pending' on an orders table (0.5% of rows)
is_deleted = 1 on a soft-delete table (0.1% of rows)
needs_review = true on a content moderation queue
retry_count > 0 on a job queue

If the rare value is more than 15-20% of the table, a regular index works fine. The partial approach shines when you’re indexing the needle, not the haystack.

Quick validation

After creating the index, always verify:

EXPLAIN ANALYZE
SELECT * FROM orders WHERE pending_id = 12345;

You should see your idx_pending_id in the output, with a low row estimate. If EXPLAIN shows a full table scan, check that you ran ANALYZE TABLE and that your WHERE clause matches the generated column - not the original status column.

If you want to take this further, combine it with covering indexes - add the columns you SELECT to the index so MySQL never touches the table at all.

Subscribe now

Found this useful? Subscribe to Podo Stack for weekly database patterns, Kubernetes internals, and Cloud Native tools ripe for production.

Still running full indexes on columns where 99% of rows don’t matter? Try the generated column approach - it takes one ALTER TABLE and the improvement shows up immediately in EXPLAIN.

eBPF Beyond Networking

Ilia Gusev — Tue, 31 Mar 2026 14:03:29 GMT

Welcome back to Podo Stack. In Issue #9 we covered Cilium - eBPF rewriting the network layer. But networking is just one tentacle. This week: five tools that push eBPF into security, profiling, and reliability.

Subscribe now

Here’s what’s good this week.

🚀 Sandbox Watch: Tetragon - Kill It at the Kernel

The problem

Your container just spawned a shell. You know about it 30 seconds later because Falco sent an alert to Slack. By then, the attacker already read /etc/shadow and opened a reverse shell. Detection is nice. Prevention is better.

The solution

Tetragon doesn’t just watch syscalls - it blocks them. Right there in the kernel. Before the call even completes.

Built by the Cilium team (Isovalent, now Cisco), Tetragon hooks into the Linux kernel via eBPF and enforces security policies at the deepest level possible. When a process violates a rule, Tetragon sends a SIGKILL before the syscall returns. No round-trip to userspace. No window for TOCTOU attacks.

And because it runs in the kernel, the overhead is below 1%. No sidecar, no agent eating your CPU.

What makes it special

Tetragon is Kubernetes-aware by default. It maps low-level kernel events to pods, namespaces, and service accounts. You write policies like “block writes to /etc for pods labeled app: frontend“ - as CRDs, not Lua scripts.

Recent additions make it even more serious: persistent enforcement continues working even if the Tetragon agent restarts. And redaction filters strip passwords and tokens from event logs right in the kernel.

The catch

Tetragon requires Linux 5.x+ with BTF support. Most modern distros have this, but old AMIs or custom kernels might not. Check before you deploy.

💎 Hidden Gem: Parca - Profile Everything, All the Time

You profile your app when something’s slow. But by the time you attach a profiler, the problem is gone. Parca flips this: it profiles your entire cluster, continuously, with less than 1% overhead.

How it works

Parca uses eBPF to sample stack traces at the kernel level. No code changes. No library imports. No restarts. It works across C, C++, Rust, Go, Python, Java, and Node.js - anything that runs on Linux.

Collected profiles go into FrostDB, a columnar database purpose-built for profile data. Then you get flame graphs, icicle charts, and - the killer feature - differential profiling. Compare Tuesday’s deployment to Monday’s. See exactly which function started eating more CPU. No guessing.

Why this matters

Traditional profiling costs 10-30% overhead. You’d never run it in production. Parca costs less than 1%, so it runs everywhere, always. When an incident hits at 3 AM, the profile data is already there.

Three practical wins: catch hotspots that waste cloud spend, find memory leaks in real time, and get instant post-mortems without “we should’ve had a profiler running.”

Parca vs Pyroscope

Pyroscope joined Grafana Labs and focuses on SDK-based profiling with broader language support. Parca stays pure eBPF - zero instrumentation, Kubernetes-native CRDs, CNCF sandbox project. If you’re already in the Cilium/eBPF world, Parca fits naturally.

⚔️ The Showdown: Falco vs Tetragon

Both watch syscalls. Both protect your cluster. But they disagree on what to do about it.

Falco is the CNCF Graduated standard for runtime threat detection. It intercepts syscalls via eBPF (or a kernel module), enriches events with Kubernetes metadata, and evaluates them against YAML rules. When something looks bad - shell in a container, unexpected network connection, read on /etc/shadow - Falco raises an alert. It’s a fire alarm.

Tetragon is a fire suppressor. Same data source (eBPF, syscalls), but instead of alerting, it kills the process in the kernel before the call finishes. No alert fatigue because the threat never completes.

Falco:
- Philosophy: Detect and alert
- Engine: Userspace rule evaluation
- Response: Alerts (Slack, PagerDuty, SIEM)
- Auto-remediation: Via Falco Talon (separate tool)
- Maturity: CNCF Graduated, 50+ output integrations

Tetragon:
- Philosophy: Enforce and prevent
- Engine: In-kernel eBPF programs
- Response: SIGKILL in kernel
- Auto-remediation: Built-in, kernel-level
- Maturity: CNCF project, Cilium ecosystem

The verdict

They’re complementary. Falco gives you the audit trail and the 50+ integrations for compliance. Tetragon gives you the kill switch. Run Falco for detection and forensics. Add Tetragon for the critical paths where “alert and hope someone notices” isn’t enough.

🔬 The Pattern: SLO from YAML with Sloth

The problem

Everyone agrees on SLOs. Nobody measures them. Why? Because writing Prometheus recording rules for SLO tracking is painful. You need multi-window burn rate calculations, error budget tracking, and alerting rules that predict budget exhaustion. That’s 50+ lines of PromQL per SLO. Per service.

The fix

Sloth takes a YAML file and generates all of it.

version: "prometheus/v1"
service: "my-api"
slos:
  - name: "availability"
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total[{{.window}}]))

Run sloth generate, apply the output, and you get:

Recording rules for burn rates across multiple time windows (5min, 30min, 1h, 2d, 6h)
Multi-burn-rate alerts that fire before your budget runs out
Grafana dashboards showing remaining error budget

99.9% availability sounds impressive until Sloth shows you that’s 43 minutes of allowed downtime per month. And your budget is burning 3x faster than expected after last Friday’s deploy.

🛠️ The One-Liner: Grafana Alloy

kubectl get pods -n monitoring -l app.kubernetes.io/name=alloy \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status'

If you’re still running Prometheus + Promtail + OTel Collector as three separate things, Grafana Alloy replaces all of them. One agent. Metrics, logs, traces, profiles.

Alloy is a Grafana distribution of the OpenTelemetry Collector with a programmable config language (think HCL, not YAML). It natively scrapes Prometheus targets, collects logs, receives OTLP traces, and - the standout feature - supports built-in clustering. Multiple Alloy instances automatically distribute scrape targets across the cluster.

It’s vendor-neutral despite the Grafana branding. Send data to Prometheus, Loki, Tempo, Mimir, or anything OTLP-compatible.

🔥 The Hot Take: The Three Pillars Are Now Four

We spent years building the “three pillars of observability” - metrics, logs, traces. And then we realized we still can’t answer “why is this function slow?”

Continuous profiling is the fourth pillar. Parca, Pyroscope, and even Grafana Cloud are betting on it. The difference: profiles show you where in the code the problem is. Not which service, not which endpoint - which function, which line.

Combine Prometheus (what’s broken), Grafana Tempo (where the request went), Loki (what happened around it), and Parca (why it’s slow at the code level). That’s full observability.

Agree? Reply and tell me if you’re already profiling in production.

Subscribe now

Questions? Feedback? Reply to this email. I read every one.

🍇 Podo Stack - Ripe for Prod.

Vertical Partitioning: Split Your Wide Table Before It Splits Your Performance

Ilia Gusev — Fri, 27 Mar 2026 07:02:17 GMT

Your users table has 45 columns. Your queries touch 5. Every time the database reads a row, it loads all 45 columns from disk into memory - the user’s login, email, and last_seen that you actually need, plus the 2KB biography, the profile picture URL, notification preferences, OAuth tokens, and 37 other columns you didn’t ask for. All of it pulled into the buffer pool, competing for cache space, slowing everything down.

Subscribe now

I hit this wall on a SaaS product with 2 million users. The user profile page loaded in 200ms. The login check - which only needed email, password_hash, and is_active - took 45ms. For a login flow that runs thousands of times per minute, 45ms felt wrong. The table had grown to 38 columns over three years of feature additions. Someone had added a bio TEXT column, a settings JSONB column, and an avatar_blob for users who uploaded directly instead of using URLs.

I split the table in two. Login dropped to 8ms. The profile page didn’t change - it was already doing a full load. But the hot path got 5x faster because the database was reading 200-byte rows instead of 2KB rows.

That’s vertical partitioning.

What vertical partitioning actually does

Horizontal partitioning splits a table into chunks by rows - users 1 through 1 million in one partition, 1 million to 2 million in another. Same columns, different rows.

Vertical partitioning splits a table by columns. Hot columns that you query constantly go into one table. Cold columns that you rarely need go into another. Same rows (same primary key), different columns.

BEFORE: one wide table
┌─────────────────────────────────────────────────────────┐
│ users (45 columns, ~2KB per row)                        │
│ id | email | password_hash | last_seen | bio | avatar | │
│    | settings | oauth_tokens | preferences | ...        │
└─────────────────────────────────────────────────────────┘

AFTER: two narrow tables
┌──────────────────────────────────┐  ┌─────────────────────────────┐
│ users (6 columns, ~200B per row) │  │ user_profiles (12 columns)  │
│ id | email | password_hash       │  │ user_id (FK) | bio          │
│    | is_active | last_seen       │  │ avatar | settings           │
│    | created_at                  │  │ oauth_tokens | preferences  │
└──────────────────────────────────┘  └─────────────────────────────┘

Why narrower rows are faster

Databases read data in pages - typically 8KB in PostgreSQL, 16KB in MySQL InnoDB. When you execute a query, the database doesn’t read individual rows. It reads entire pages into memory.

A wide row (2KB) fits about 4 rows per 8KB page. A narrow row (200 bytes) fits about 40 rows per page. That’s 10x more rows per I/O operation.

This has cascading effects:

Buffer pool efficiency. Your database caches pages in memory. If each page holds 40 rows instead of 4, the same amount of RAM caches 10x more rows. Cache hit rates go up. Disk reads go down.Sequential scan speed. When PostgreSQL does a sequential scan on a narrow table, it reads fewer pages to cover the same number of rows. A full scan of 2 million narrow rows might read 50K pages. The same scan on the wide table reads 500K pages.

Index performance. In PostgreSQL, the heap (table data) is separate from indexes. An index lookup finds the row location, then fetches the row from the heap. With narrower rows, the heap is smaller, so even random heap fetches are more likely to hit cached pages.

Implementation: step by step

Let’s walk through splitting a real users table.

1. Identify hot and cold columns

Look at your actual queries. Not what you think runs, but what really runs.

-- PostgreSQL: find which columns are actually used
-- Check pg_stat_statements or your query logs
-- Look for patterns like:
SELECT email, password_hash, is_active FROM users WHERE email = $1;  -- login: HOT
SELECT last_seen FROM users WHERE id = $1;                           -- presence: HOT
SELECT bio, avatar, settings FROM users WHERE id = $1;               -- profile page: COLD

Hot columns are touched by high-frequency queries - authentication, authorization, listings, search results. Cold columns are touched by low-frequency queries - profile detail pages, settings screens, admin views.

2. Create the cold table

-- Create the details table with the same primary key
CREATE TABLE user_profiles (
    user_id     BIGINT PRIMARY KEY REFERENCES users(id),
    bio         TEXT,
    avatar      BYTEA,
    settings    JSONB DEFAULT '{}',
    oauth_tokens JSONB,
    preferences JSONB DEFAULT '{}',
    company     VARCHAR(200),
    location    VARCHAR(200),
    website     VARCHAR(500),
    twitter     VARCHAR(100),
    github      VARCHAR(100),
    linkedin    VARCHAR(100)
);

3. Migrate the data

-- Copy cold columns to the new table
INSERT INTO user_profiles (user_id, bio, avatar, settings, oauth_tokens,
    preferences, company, location, website, twitter, github, linkedin)
SELECT id, bio, avatar, settings, oauth_tokens,
    preferences, company, location, website, twitter, github, linkedin
FROM users;

-- Verify row counts match
SELECT COUNT(*) FROM users;          -- 2,000,000
SELECT COUNT(*) FROM user_profiles;  -- 2,000,000

4. Drop cold columns from the main table

-- Remove cold columns from the hot table
ALTER TABLE users
    DROP COLUMN bio,
    DROP COLUMN avatar,
    DROP COLUMN settings,
    DROP COLUMN oauth_tokens,
    DROP COLUMN preferences,
    DROP COLUMN company,
    DROP COLUMN location,
    DROP COLUMN website,
    DROP COLUMN twitter,
    DROP COLUMN github,
    DROP COLUMN linkedin;

In PostgreSQL, DROP COLUMN doesn’t physically remove the data immediately - it marks the column as dropped. Run VACUUM FULL users to reclaim the space (this locks the table, so schedule it during maintenance).5. Create a view for backward compatibility

-- Applications that SELECT * still work
CREATE VIEW users_full AS
SELECT u.*, p.bio, p.avatar, p.settings, p.oauth_tokens,
    p.preferences, p.company, p.location, p.website,
    p.twitter, p.github, p.linkedin
FROM users u
LEFT JOIN user_profiles p ON u.id = p.user_id;

The LEFT JOIN ensures users without a profile row still appear. Existing queries that reference the old wide table can switch to the view without code changes - at least as a migration bridge.

Handling writes

This is where vertical partitioning gets tricky. Before the split, inserting a new user was one statement. Now it’s two, and they need to be atomic.

-- Must be in a single transaction
BEGIN;

INSERT INTO users (email, password_hash, is_active, created_at)
VALUES ('new@example.com', '$2b$12$...', true, NOW())
RETURNING id;

-- Use the returned id
INSERT INTO user_profiles (user_id, bio, settings)
VALUES (currval('users_id_seq'), '', '{}');

COMMIT;

If your application uses an ORM, you’ll need to update the model layer. In most ORMs, this means creating a separate UserProfile model with a one-to-one relationship. It’s more code, but it’s straightforward.

For updates, the same rule applies - if you’re updating columns in both tables, wrap it in a transaction:

BEGIN;
UPDATE users SET last_seen = NOW() WHERE id = 42;
UPDATE user_profiles SET bio = 'Updated bio' WHERE user_id = 42;
COMMIT;

In practice, most updates only touch one table. The login flow updates last_seen in users. The profile edit updates bio in user_profiles. Cross-table updates are the exception, not the rule.

Common use cases

User profiles. The example above. Authentication data (hot) vs display data (cold). Almost every application with user accounts benefits from this.

Product catalogs. Core product data - id, name, price, category, is_available - queried on every search result and listing page. Extended data - full_description TEXT, specifications JSONB, manufacturer_details - only loaded on the product detail page.

Sensitive data isolation. PII columns (SSN, date of birth, bank account) in a separate table with stricter access controls, encryption at rest, and audit logging. The main table doesn’t even contain the sensitive data, so a breach of the main table leaks less.

Audit columns. created_by, updated_by, approved_by, audit_notes - columns that exist for compliance but aren’t used in normal application queries. Moving them out keeps the hot table lean.

When NOT to do this

Small tables. If your table has 10,000 rows and fits entirely in memory regardless of width, vertical partitioning adds complexity for zero performance gain. Don’t optimize what isn’t slow.

All columns are hot. If your queries genuinely use most columns most of the time, splitting the table just adds JOINs. Vertical partitioning only helps when there’s a clear hot/cold divide.

Write-heavy workloads with cross-column updates. If every write touches columns in both tables, you’re doubling your write I/O and adding transaction coordination overhead. The read savings might not offset the write cost.

Measuring the impact

Before and after, check these:

-- PostgreSQL: table size before and after
SELECT pg_size_pretty(pg_total_relation_size('users'));

-- Average row width
SELECT avg(pg_column_size(t.*)) FROM users t LIMIT 10000;

-- Buffer cache hit ratio
SELECT
    sum(heap_blks_hit) AS cache_hits,
    sum(heap_blks_read) AS disk_reads,
    round(sum(heap_blks_hit)::numeric /
        (sum(heap_blks_hit) + sum(heap_blks_read)), 4) AS hit_ratio
FROM pg_statio_user_tables
WHERE relname = 'users';

If your cache hit ratio goes from 85% to 98% after the split, you’ve freed up buffer pool space that every query on the system benefits from. The impact often extends beyond the table you partitioned.

Found this useful? Subscribe to Podo Stack for weekly database patterns and Cloud Native tools ripe for production.

Subscribe now

Have you split a wide table in production? What was the before/after difference? I’d love to hear your numbers - reply to this email or leave a comment.

Change Data Capture: Stop Copying 50M Rows to Move 5K Changes

Ilia Gusev — Wed, 25 Mar 2026 07:01:53 GMT

Your ETL copies 50 million rows every night. Only 5,000 changed. The other 49,995,000 rows are identical to yesterday. You’re burning compute, network, and storage to move data that hasn’t moved.

Subscribe now

I’ve seen this pattern at three different companies. The nightly batch job runs for 4 hours, the data warehouse is “fresh” by 6 AM, and everyone accepts this as normal. It isn’t. Change Data Capture (CDC) exists specifically to solve this - track only what changed, move only the deltas. The difference between copying a full table and streaming 5K changes is the difference between hauling the entire library and borrowing one book.

What CDC actually does

CDC captures INSERT, UPDATE, and DELETE events as they happen (or shortly after) and delivers them downstream. Instead of asking “what does the table look like right now?” you ask “what changed since last time I checked?”

The result is the same - your target stays in sync with the source. But the cost drops dramatically. Less data transferred, less compute on both sides, and the target can be minutes behind the source instead of hours.

There are three main approaches, each with different trade-offs.

Method 1: Timestamp-based

The simplest form. Add created_at and updated_at columns to your table, then query for rows modified since the last sync.

-- Source table
CREATE TABLE orders (
    id          BIGINT PRIMARY KEY,
    customer_id INT,
    amount      DECIMAL(10,2),
    status      VARCHAR(20),
    created_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

-- CDC query: get everything that changed since last run
SELECT *
FROM orders
WHERE updated_at > '2026-03-07 00:00:00';

Your ETL runs this query every 15 minutes, processes the results, and records the latest updated_at as the watermark for the next run.

Pros: Dead simple. No special infrastructure. Works with any database. You can implement it in an afternoon.

Cons: It misses hard deletes. If someone runs DELETE FROM orders WHERE id = 42, there’s no row left to have an updated_at. The row just disappears, and your target never finds out. You’d need a soft-delete pattern (is_deleted flag) or periodic full reconciliation to catch these.

It also struggles with bulk updates that don’t touch the timestamp column. If someone does ALTER TABLE orders ADD COLUMN priority INT DEFAULT 0, every row now has the same value but updated_at didn’t change. Schema changes are invisible.

Good for: simple sync scenarios where deletes are rare and you control the source schema.

Method 2: Trigger-based

Database triggers fire on every INSERT, UPDATE, and DELETE and write the change to a shadow table.

-- Shadow table to capture changes
CREATE TABLE orders_changes (
    change_id   BIGSERIAL PRIMARY KEY,
    operation   CHAR(1),  -- 'I', 'U', 'D'
    changed_at  TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    order_id    BIGINT,
    customer_id INT,
    amount      DECIMAL(10,2),
    status      VARCHAR(20)
);

-- Trigger function (PostgreSQL)
CREATE OR REPLACE FUNCTION capture_order_changes()
RETURNS TRIGGER AS $$
BEGIN
    IF TG_OP = 'DELETE' THEN
        INSERT INTO orders_changes (operation, order_id, customer_id, amount, status)
        VALUES ('D', OLD.id, OLD.customer_id, OLD.amount, OLD.status);
        RETURN OLD;
    ELSE
        INSERT INTO orders_changes (operation, order_id, customer_id, amount, status)
        VALUES (
            CASE WHEN TG_OP = 'INSERT' THEN 'I' ELSE 'U' END,
            NEW.id, NEW.customer_id, NEW.amount, NEW.status
        );
        RETURN NEW;
    END IF;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER orders_cdc_trigger
AFTER INSERT OR UPDATE OR DELETE ON orders
FOR EACH ROW EXECUTE FUNCTION capture_order_changes();

Your ETL reads from orders_changes, processes the deltas, and deletes or marks the consumed rows.

Pros: Catches everything - inserts, updates, and deletes. No row is missed. You get the before and after state if you capture both OLD and NEW.

Cons: Triggers are synchronous. Every INSERT INTO orders now also does an INSERT INTO orders_changes within the same transaction. On a table with 10,000 writes per second, that’s 10,000 extra inserts. Your OLTP workload just got heavier.

Triggers are also database-specific. The PostgreSQL trigger above won’t work on MySQL or SQL Server. You’re writing and maintaining CDC logic per database engine. And triggers can be fragile - they silently break when someone alters the table structure without updating the trigger.

Good for: moderate-write workloads where you need to catch deletes and don’t want external tooling.

Method 3: Log-based (the gold standard)

Every transactional database maintains a write-ahead log (WAL in PostgreSQL, binlog in MySQL, redo log in Oracle, transaction log in SQL Server). This log records every change before it hits the actual data files. It’s how databases survive crashes - replay the log, recover the state.

Log-based CDC reads this log directly. No queries against the table, no triggers, no impact on OLTP performance. The log is already being written - CDC just tails it.

PostgreSQL WAL → Debezium → Kafka → your data warehouse
MySQL binlog   → Debezium → Kafka → your data warehouse
SQL Server CDC → Fivetran  → Snowflake

Debezium is the most popular open-source tool for this. It connects to the database’s replication slot (PostgreSQL) or binlog (MySQL), reads every committed transaction, and produces change events as JSON:

{
  "op": "u",
  "before": {"id": 42, "status": "pending", "amount": 99.99},
  "after":  {"id": 42, "status": "shipped", "amount": 99.99},
  "source": {
    "db": "production",
    "table": "orders",
    "ts_ms": 1709827200000
  }
}

You get the operation type (c = create, u = update, d = delete, r = read/snapshot), the before state, the after state, and metadata about where the change came from. Everything you need to replay the change downstream.

Pros: Asynchronous - zero impact on OLTP performance. Catches everything including DDL changes. Near real-time (seconds of latency, not hours). Works at any scale because you’re reading a sequential log, not scanning a table.

Cons: More infrastructure to run. Debezium needs Kafka (or Kafka Connect), you need to manage connectors, handle schema evolution, monitor lag. It’s not an afternoon project.

The database also needs to be configured for logical replication (PostgreSQL) or row-based binlog (MySQL). In managed services like RDS or Cloud SQL, this is usually a checkbox. On-premise, it might mean changing database parameters and restarting.

Good for: anything in production. If you’re serious about keeping a data warehouse or downstream service in sync, log-based CDC is the answer.

Comparing the three

Timestamp-based: Doesn’t catch deletes. Adds query load to OLTP. Minutes of latency. Low setup complexity. Can’t catch DDL changes. Medium scale ceiling. Tools: any SQL client.

Trigger-based: Catches deletes. Adds write overhead. Seconds of latency. Medium complexity. Can’t catch DDL. Medium scale ceiling. Tools: DB-native triggers.

Log-based: Catches deletes. Zero OLTP impact. Seconds of latency. High setup complexity. Catches DDL changes. High scale ceiling. Tools: Debezium, Fivetran, Striim.

CDC and Slowly Changing Dimensions

If you’re building a data warehouse, CDC pairs naturally with Slowly Changing Dimensions. CDC tells you what changed. SCD defines how to store that change in your dimension tables.

A Type 2 SCD workflow with log-based CDC looks like this:

Debezium captures an update: customer moved from New York to Austin
Your ETL receives the change event
It closes the current dimension row (valid_to = today, is_current = false)
It inserts a new row for the Austin version (valid_from = today, is_current = true)

Without CDC, you’d scan the entire customer table every night to detect this one address change. With CDC, the change arrives within seconds and your dimension is always current.

Getting started

If you’re running on timestamp-based sync and it works - don’t fix what isn’t broken. Add it to your migration backlog and prioritize based on pain.

If you’re feeling the pain - stale data, missed deletes, long batch windows - here’s the progression:

Start with timestamp-based if you just need to prototype. Add updated_at columns, write a simple sync script, validate the approach.
Move to log-based when you need reliability. Set up Debezium with Kafka Connect, point it at your database, and let it stream changes. Skip the trigger phase - it adds complexity without the benefits of log-based CDC.
Monitor lag. CDC is only useful if the stream keeps up with the source. Track consumer lag in Kafka, alert on it, and scale consumers if needed.

The tools are mature. Debezium has been in production at thousands of companies for years. Fivetran and Striim handle the infrastructure for you if you don’t want to run Kafka yourself. The hard part isn’t the tooling - it’s deciding to stop copying 50 million rows when you only need 5,000.

Found this useful? Subscribe to Podo Stack for weekly data engineering patterns and Cloud Native tools ripe for production.

Subscribe now

Already using CDC in production? Still on nightly batch ETL? I’d love to hear what your setup looks like - reply to this email or leave a comment.

Your Infrastructure Has an API Now, and It's Not Terraform

Ilia Gusev — Tue, 24 Mar 2026 07:02:38 GMT

In Issue #4, we talked about Platform Engineering and briefly mentioned Crossplane as a Terraform alternative. Time to go deep. What if your infrastructure had a real API - not a CLI tool you run from a CI pipeline, but a Kubernetes controller that never stops reconciling?

Subscribe now

That’s exactly what Crossplane does. And if you’re still wrapping Terraform in CI jobs, you might be solving 2025’s problems with 2016’s architecture.

The Pattern: Crossplane as Cloud Control Plane

Your infrastructure is a Kubernetes resource now.

Crossplane extends the K8s API with CRDs for cloud resources. RDS instances, S3 buckets, VPCs, IAM roles - all of them become objects in your cluster. You kubectl apply them just like you would a Deployment or a ConfigMap.

This sounds like Terraform with extra steps. It’s not. The difference is fundamental. Terraform runs as a CLI - you execute terraform apply, it creates resources, writes state to a file, and walks away. If someone modifies the resource manually in the console? Terraform doesn’t know until you run terraform plan again. That’s drift, and it’s your problem.

Crossplane runs reconciliation controllers. Continuously. Every 10 minutes (configurable), it checks actual cloud state against desired state and fixes any drift. No human intervention. No CI job. It just works - exactly like how a Deployment controller makes sure your pods are always running.

The architecture has four layers. A Provider is a controller that talks to a specific cloud API (AWS, GCP, Azure). An XRD (Composite Resource Definition) is the schema for your custom abstraction. A Composition is the implementation - it maps your abstraction to actual cloud resources. And a Claim is what developers interact with.

Here’s the workflow. A developer runs kubectl apply -f postgres-claim.yaml. Crossplane creates an RDS instance, attaches security groups, configures parameter groups, and writes the connection credentials into a Kubernetes Secret in the developer’s namespace. One YAML file. No Terraform, no tickets, no waiting.

Crossplane graduated in the CNCF in 2024. It’s not an experiment anymore.

Hidden Gem: Composition Functions

When YAML patching isn’t enough, bring code.

Standard Crossplane Compositions use patch-and-transform - you map fields from the Claim to the managed resources. It works, but it’s limited. No loops. No conditionals. No “create this resource only if the environment is production.”

Composition Functions change everything. Instead of static YAML patching, your Composition runs a pipeline of gRPC functions. Each function is a container - written in Go, Python, whatever you like - that receives the composite resource and desired state, then returns modified desired state.

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
spec:
  mode: Pipeline
  pipeline:
    - step: create-resources
      functionRef:
        name: function-go-templating
    - step: add-tags
      functionRef:
        name: function-auto-tagger
    - step: validate
      functionRef:
        name: function-compliance-check

This unlocks things that were impossible before. Dynamic naming based on environment. Conditional resource creation. External API calls during rendering - like fetching the latest AMI ID or checking a CMDB. You’re writing real logic, not fighting YAML indentation.

The Showdown: Crossplane vs Terraform

Terraform: CLI-driven (terraform apply). State in .tfstate file - drift possible. Manual drift detection (terraform plan). Self-service requires CI/CD wrapper. Abstraction via modules. Secrets live inside .tfstate.

Crossplane: Continuous reconciliation loop. State in K8s etcd - auto-healing. Automatic drift detection via controller. Native self-service through K8s RBAC + Claims. Abstraction via XRD + Compositions. Secrets in K8s Secrets + Vault integration.

The verdict: Terraform isn’t going away. It’s still great for bootstrapping clusters, managing DNS zones, and one-off infrastructure. But for day-2 operations - the “developers need databases and don’t want to file tickets” problem - Crossplane fits the Kubernetes-native model better. Continuous reconciliation beats run-and-pray.

Deep Dive: Crossplane + Kyverno = Safe Self-Service

Claims are K8s resources. Kyverno can validate them.

Here’s where it gets interesting. A Claim is just a Kubernetes custom resource. That means every policy tool that works with K8s admission control - Kyverno, OPA Gatekeeper, Kubewarden - works with Claims out of the box.

Want to limit dev namespaces to db.t3.small instances with max 100GB storage? Kyverno ClusterPolicy. Want to auto-add billing tags and cost-center labels to every Claim? Kyverno mutation rule. Want to block db.r6g.16xlarge in anything that isn’t the production namespace? Validation policy.

The beauty is layered enforcement. Validate at the Claim level - that’s the fastest feedback loop, developers see errors immediately on kubectl apply. Then validate again at the MR (Managed Resource) level as a safety net. Two layers, same tool.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: limit-rds-size
spec:
  rules:
    - name: restrict-instance-class
      match:
        resources:
          kinds:
            - PostgreSQLInstanceClaim
      validate:
        message: "Dev namespaces limited to db.t3.small"
        pattern:
          spec:
            parameters:
              instanceClass: "db.t3.small"

Self-service without guardrails is just chaos with a UI. Crossplane + Kyverno gives you both.

The One-Liner: Secret Management Chain

kubectl get secret my-app-db-conn -n my-app -o jsonpath='{.data.endpoint}' | base64 -d

Where did that secret come from? Nobody created it manually. Here’s the chain.

When you define a Claim with writeConnectionSecretToRef, Crossplane creates the cloud resource (say, an RDS instance), reads the generated credentials from the cloud API, and writes them into a Kubernetes Secret in the developer’s namespace. The developer’s app mounts that secret - no copy-pasting passwords, no Slack DMs with credentials.

The chain goes deeper. The Managed Resource (MR) writes its secret. The Composite Resource (XR) aggregates secrets from multiple MRs. The Claim exposes the final secret to the developer’s namespace. Three levels, fully automated.

For production, you don’t want raw K8s Secrets. Crossplane’s StoreConfig integrates with HashiCorp Vault, AWS Secrets Manager, or any external secret store. The developer’s workflow doesn’t change - they still reference a secret name in their Claim. Where it’s stored is an infrastructure concern, not a developer concern.

Podo Stack

Firecracker: the minimalism that runs your Lambda function

The dilemma that didn't have a clean answer

What got cut

Links

The architecture that makes sub-second boot work

Where Firecracker actually runs in production

Links

Where Firecracker is the wrong tool

The comparison that matters for platform decisions

Links

The operational pieces that aren't in the Getting Started docs

Links

Summary

Podo #016: Kyverno Isn't Security. It's Governance - And You're Using 20% of It.

Mutating admission is where the real power lives

Most teams reach for Validate when Mutate would be safer.

Links

Generate rules make cross-namespace propagation trivial

Stop reaching for reflector-style third-party tools.

Links

Cleanup policies plus kor: making orphans visible

Governance is also about what leaves the cluster.

Links

CronJob has seven failure modes nobody warned you about

The quietest workload in Kubernetes has the loudest silent failures.

Links

The image hygiene pipeline

Seven vendor-neutral tools, one lifecycle.

Links

Closing

Server-Side Apply and managedFields: the Kubernetes feature that ends controller fights

What managedFields actually contains

Links

The conflict a client-side apply user never saw

Force, conflict, and what actually happens on collision

Links

List merge strategies: where subtle bugs live

Links

How ownership transfers happen

The operator pattern that gets SSA right

Links

When you still want client-side apply

Practical diagnostics

Summary

Prometheus WAL: the silent disk growth nobody diagnoses correctly

What the WAL actually contains

Links

Ingestion through the WAL: the write path

Checkpointing: the mechanism that makes WAL bounded

Cardinality as a WAL problem, not a memory problem

Links

Why your Prometheus took 22 minutes to start

Links

WAL corruption: what actually happens

Disk layout that actually matters

The three signals to actually alert on

Links

Summary

Podo #015: The Quietest Performance Bug: What Happens in a Pod's First 60 Seconds

cgroup v2 silently ate 61% of your CPU priority

The linear formula was the wrong shape.

Links

Four ways to kill image pull delay

One problem, four philosophies, and they compose.

Links

The scheduler can't see the neighbor stealing your latency

Cache contention lives in a blind spot.

Links

A pentester's view of your brand-new pod

The same 60 seconds, read from the other side.

Links

Closing

HDFS Lost. How Object Storage and Table Formats Won the Data Lake

The collapse of data locality

The coupling problem

The rename problem and why it didn’t matter

What won

When HDFS still makes sense

The wider lesson

🔥 The Trap: Topology Spread Constraints with `DoNotSchedule`