How to Save Memory in a Consistent Hash Ring, Cloudflare Style

A practical guide to optimize a consistent hash ring and reduce its memory footprint without losing load balancing, inspired by Cloudflare and Pingora's case.

Software engineering

Consistent hashing distributes requests among servers with minimal reassignments when the server set changes. Cloudflare uses it in its Pingora Backend Router to distribute cacheable requests, and they documented how they reduced the memory footprint of that service by over 100 TB by fine-tuning their Rust implementation. Here, we translate that idea into a method you can apply to your own ring.

In 30 seconds Each server occupies many points in the ring. Cloudflare used probability to calculate how many are actually needed, cut 90%, and compacted each entry from 8 to 6 bytes. Then they deployed it gradually to avoid emptying the cache.

What you need

You need access to your ring's implementation (in Cloudflare's case, the pingora-ketama crate), the actual list of backends with their weights, and a way to measure the load distribution you currently get. Without that baseline, you won't know if you've improved or worsened.

It's also helpful to understand the birthday problem: the more 32-bit points you place, the more likely it is that two will coincide, and those collisions add error to the distribution. This is why "more hashes" isn't free or infinitely better.

Each server appears many times in the ring; the fewer points, the less memory.

The step-by-step method

1. Measure the current distribution error: how much load each backend receives compared to what it should receive based on its weight.
2. Note how many hashes per server you use and how many rings you maintain (weights and feature combinations multiply them).
3. Simulate the distribution with fewer hashes per server and plot the resulting error. You'll see that each improvement in precision requires almost an order of magnitude more points.
4. Choose the minimum that meets your acceptable error margin. Cloudflare found that the last 90,000 hashes only improved the error by 0.7%.
5. Compact the structure: if the hash and server index fit in 6 bytes, don't store 8.
6. Deploy the new ring in parallel with the old one and gradually shift traffic.

// Idea, not Cloudflare's code:
// before: 8 bytes per point
struct PuntoViejo { hash: u32, servidor: u32 }

// after: 6 bytes (32-bit hash + 16-bit index)
#[repr(packed)]
struct PuntoCompacto { hash: u32, servidor: u16 }

This snippet is merely an illustration of the compaction described in the article; the 16-bit index assumes you have fewer than 65,536 servers, so validate that limit in your case. The full details are in the InfoQ report and the original blog post.

Beware of the cache

Changing the ring changes which server each cacheable request goes to. Even if the new ring is better, switching the entire network at once would invalidate almost all cached content and spike traffic to the origins. That's why Cloudflare deployed it alongside the existing ring, progressively, to limit this rotation.

A staggered migration avoids the peak traffic to the origins.

Tip Before touching anything, take a memory profile and locate which structure dominates. Cloudflare found the problem because they investigated the consumption of their hash structures, not because they guessed. And don't assume 90% is universal: your number depends on your acceptable error and your weights.

If you want to see how other developers feel about why consistent hashing is still useful despite its cost, there's a discussion on Hacker News with interesting viewpoints.

Opinion · The number that worries me

The figure that worries me is that Cloudflare cut nearly 90% of hashes per server, and even so, the last 90,000 hashes only improved the error by 0.7%. This means that for a long time, an enormous amount of memory was paid for an almost invisible improvement.

It worries me how many systems, without as much scale or as large a team behind them, carry inherited parameters that no one has ever measured. A number chosen by custom multiplies for each server and for each ring.

Related articles from LaRebelión

How to Scale Git: Reference Quorums and Object StorageCloudflare's tsrs: A Guide to Evaluating the TypeScript to Rust PortHow to Prepare Your Open Source Project for Anthropic's OSS Scanner

• Software engineering
• Guide to Migrating to Final Structured Concurrency in JDK 28
• How to Evaluate Pyrefly in Your Python Project Before Dropping Pytype or Mypy

Original Source: infoq.com

Article generated with AI.larebelion

Kernel · Head of Technology · Spain

«Cloudflare not only saved 100 TB: it showed that an inherited parameter can cost more than any new algorithm.»