Redis memory: why 16 GB of data needs 32 GB of RAM

Fragmentation, RSS, jemalloc, eviction, fork overhead

The box had 32 GB. The dashboard said Redis was holding 16 GB of data, comfortably under half. Then the OOM killer took it at 03:00, and the on-call engineer spent the first twenty minutes convinced the graph was lying, because how does a process reported at 16 GB get killed on a 32 GB machine. The graph wasn't lying. It was just answering a different question than the one being asked. used_memory was 16 GB. The resident set the kernel actually accounted for was sitting near 31 GB, and Redis had no idea, because the bytes it gave back to its allocator never made it back to the OS. This is the single most expensive misunderstanding about running Redis at scale, and it applies one-for-one to Valkey, which forked from the same codebase and inherited the same allocator behavior.

Two numbers that should be one

Redis reports memory through INFO memory, and the first thing to internalize is that the two headline numbers measure different layers of the stack. used_memory is what the allocator handed Redis for its data and bookkeeping - the logical footprint. used_memory_rss is the resident set size the operating system sees, the physical pages actually mapped to the process. In a perfect world they'd track each other. They don't, and the gap is where the trouble lives.

The ratio between them, mem_fragmentation_ratio, is used_memory_rss divided by used_memory. People read it as a fragmentation gauge and mostly that's right, but it's quietly overloaded. A value around 1.0 to 1.1 is healthy. Up around 1.5 means roughly half again as much physical memory as your data needs, which is what bit the box above. The part that catches teams off guard is the other direction. A ratio below 1.0 is not "great, negative fragmentation." It means part of Redis has been pushed to swap, so the OS-resident portion is now smaller than the logical data, and the difference is sitting on disk. That's the worst state on this whole page. A swapping Redis has tail latencies that look like the network is on fire, and a fragmented Redis just wastes RAM. I'll take the wasted RAM.

One more number worth pulling: mem_fragmentation_bytes, the absolute gap. On a small instance a ratio of 1.4 is a rounding error in bytes; on a 60 GB instance it's a second machine's worth of memory you're paying for and not using.

Why the allocator keeps what you delete

The reason RSS doesn't fall when you delete keys comes down to jemalloc, the allocator Redis and Valkey ship with by default. jemalloc doesn't hand out memory in the exact size you ask for. It rounds every request up into one of a fixed set of size classes - 8, 16, 32, 48, 64 bytes and so on up through larger spacings. Ask for a 33-byte value and you get a 48-byte slot, and those 15 bytes are gone to internal fragmentation before any key has been deleted. Across millions of values of awkward sizes, that overhead alone is real money.

The deletion problem is worse and less intuitive. jemalloc groups allocations of the same size class onto runs of contiguous pages. When you delete a key, its slot goes back onto jemalloc's free list for that size class - available for the next value of that size, but still resident, still counted in RSS. The OS only gets a page back when an entire run of pages becomes free and jemalloc decides to return it. If one live 64-byte object remains on a page full of freed 64-byte slots, that whole page stays mapped. So a workload that fills to 30 GB, deletes 90% of its keys, and sits there will keep reporting an RSS near 30 GB while used_memory drops to 3 GB. The data is gone. The memory isn't. This is exactly why restarting Redis "fixes" a memory problem - the restart rebuilds the heap from scratch with no holes in it, and for a while the ratio is beautiful again. It's not a fix, it's a reset, and the fragmentation comes right back as the access pattern resumes.

Active defragmentation, and what it costs

Redis 4.0 added activedefrag precisely for the stuck-RSS case. With jemalloc compiled in, Redis can ask the allocator which objects are sitting on sparsely-used pages and then copy those live objects to freshly allocated, densely-packed memory, freeing the old pages back to the OS. It runs incrementally in a background cycle so it doesn't stall the main thread, governed by thresholds like active-defrag-ignore-bytes and active-defrag-threshold-lower, and a CPU ceiling so it backs off under load.

It works, but it isn't free and it isn't instant. Defrag burns CPU walking and relocating objects, and on a busy instance you'll watch the fragmentation ratio drift down over minutes to hours rather than snap. I've seen people enable it, check the ratio thirty seconds later, see no change, and conclude it's broken. Give it time. There's also a manual lever, MEMORY PURGE, which tells jemalloc to release as much free-but-resident memory as it can right now - useful as a one-shot after a big mass deletion, but it doesn't relocate live objects the way active defrag does, so it can't unstick a page that still holds one survivor.

The fork that doubles you

Here's the operational landmine that turns a comfortable instance into an OOM kill. Redis persists with RDB snapshots and AOF rewrites, and both work by calling fork(). The child gets a copy-on-write view of the parent's memory: pages are shared until one side writes to them, at which point the kernel copies that page so the two processes diverge. If your instance is idle during the save, COW costs almost nothing. If your instance is taking writes during the save - and a production cache always is - every page the parent touches gets duplicated, and in the pathological case the child ends up holding a near-complete second copy of the dataset.

That's the real shape of "16 GB needs 32 GB." It isn't just fragmentation. A 16 GB dataset under steady write load, caught mid-RDB-save, can briefly push total RSS toward 32 GB between parent and child. If you sized the box for the dataset plus a fragmentation margin and forgot the fork, the save itself is what tips you into the killer. This is why vm.overcommit_memory = 1 is in every Redis production guide - without it, the fork() can be refused outright when the kernel does naive accounting, and your background save fails. The deeper fix is to leave real headroom: plan for the dataset, plus fragmentation, plus a fork copy under your peak write rate.

maxmemory, eviction, and the OOM that comes back as an error

If you don't set maxmemory, Redis grows until the kernel kills it. So you set it. But maxmemory plus the wrong policy trades one outage for another. The default policy is noeviction: at the limit, reads still work, but any write that would grow memory gets rejected with an OOM error returned to the client. For a cache fronting a database, that's usually catastrophic - the app starts taking write errors under exactly the load spike that filled the cache, and a "caching layer" becomes a hard dependency that fails closed.

The eviction policies are the alternative, and which one fits depends on what the instance is for. allkeys-lru and the newer allkeys-lfu evict across the whole keyspace by approximate recency or frequency - the right default for a pure cache where everything is disposable. The volatile-* family (volatile-lru, volatile-lfu, volatile-ttl, volatile-random) only evicts keys that carry a TTL, which is what you want when the instance mixes throwaway cache entries with keys that must persist. The trap in the volatile policies is that if nothing currently has a TTL, there's nothing eligible to evict, and you're back to OOM errors despite having "configured eviction." LRU and LFU here are approximations, by the way - Redis samples a handful of keys rather than maintaining a true global ordering, because exact LRU across millions of keys would cost more memory than it saves.

Eviction and fragmentation interact in a way that surprises people. maxmemory is enforced against used_memory, the logical number, not against RSS. So an instance can be evicting hard - throwing away useful data to stay under the limit - while RSS sits well above maxmemory because of fragmentation holes the allocator won't release. You're losing cache hits and over-consuming RAM at the same time. When you see eviction counts climbing on an instance that the dashboard says has spare physical memory, fragmentation is usually the reason the two views disagree.

Big keys and why you never run KEYS

A single hash, set, or sorted set with millions of members is its own memory hazard. It allocates as one logical structure, it can't be partially evicted - eviction works at the key granularity, so a 4 GB sorted set is all-or-nothing - and deleting it synchronously blocks the main thread while Redis frees every member, which is why UNLINK (background free) exists alongside DEL. Big keys also wreck the COW math during forks, because one write into a giant value can dirty a lot of pages at once.

Finding them is where the second classic outage hides. The instinct is KEYS * to list the keyspace, and KEYS is O(n) and blocks the single main thread for the entire scan. On a large instance that's a multi-second stall where Redis answers nothing, and you've created an incident while investigating one. SCAN is the answer - a cursor-based iterator that returns small batches and lets other commands interleave. Same rule for the type-specific variants, HSCAN, SSCAN, ZSCAN, when you're digging into one big collection. For sizing, redis-cli --bigkeys and --memkeys walk the keyspace with SCAN under the hood and report the largest keys per type without freezing anything.

The decision framework

The numbers tell you which problem you have. Read them in this order before you touch a config.

Start with the fragmentation ratio. If it's below 1.0, stop everything else - you're swapping, and the fix is more RAM or a smaller dataset, not defrag. If it's comfortably between 1.0 and 1.3 and RSS fits the box with fork headroom, you don't have a memory problem and you should resist the urge to tune. If it's elevated, 1.4 and up, and the absolute mem_fragmentation_bytes is large enough to matter, that's when activedefrag yes earns its CPU.

Size the box for three things stacked, not one. The dataset is the floor. Add a fragmentation margin on top - 20 to 50% depending on how churn-heavy and awkwardly-sized your values are. Then add room for a fork copy proportional to your write rate during saves. A 16 GB dataset on a 16 GB box is a future incident with a date on it.

Pick the eviction policy from what the instance actually is. Pure disposable cache: allkeys-lru or allkeys-lfu, and never noeviction. Mixed cache-and-state where some keys must survive: a volatile-* policy, but only if you're disciplined about setting TTLs, because an empty eligible set lands you back on OOM errors. Durable store you don't want silently shrinking: noeviction on purpose, with hard alerting on used_memory so a human intervenes before the writes start failing.

Common mistakes

The same handful of errors show up across nearly every Redis memory incident.

• Sizing the instance for used_memory and ignoring RSS. The kernel kills on RSS, and RSS carries fragmentation plus any fork copy. The logical number is the smallest of the three.

• Reading a sub-1.0 fragmentation ratio as good news. It means swap, which is the worst outcome on this page, and it disguises itself as a low number.

• Running noeviction on a cache. The first traffic spike that fills it turns the cache into a write-rejecting hard dependency, and the database behind it loses the buffer right when it needs it most.

• Setting a volatile-* policy and then not setting TTLs. Nothing is eligible to evict, so the instance OOMs anyway and the eviction config reads like a safety net that was never connected.

• Forgetting the fork. The instance runs fine for weeks, then OOM-kills exactly during an RDB save or AOF rewrite under write load, and the postmortem blames "a memory leak" instead of copy-on-write.

• Running KEYS * to find big keys. O(n) on the main thread, a multi-second stall, an outage created by the investigation. SCAN and --bigkeys exist for this.

• Restarting to "fix" memory and calling it done. The restart resets the heap and the ratio looks great, but the same access pattern rebuilds the same fragmentation, and you're back in a week without having changed anything.

• Enabling activedefrag and judging it in thirty seconds. It works over minutes to hours and costs CPU while it runs. Watch the ratio trend, not a single sample.

None of this changes whether you're on Redis or Valkey - same allocator, same fork, same eviction model, same numbers in INFO. The mental model that saves you is small and unglamorous: there are three memory figures, not one, the OS only cares about the largest, and the gap between logical and resident is paid for in real RAM whether or not anything is using it.