Running Memcached?

Memcached hit rate or slab classes acting up? We fix it.

Hit rate has dropped from the high nineties to the low eighties over the quarter, a couple of slab classes are evicting on every write while others sit half empty, and large keys have started hitting the one-megabyte item ceiling. We open stats slabs, measure what the cache is actually holding, and bring the hit rate and the evictions back to numbers that help the database instead of chasing it.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

memcached.log

still broken

What it is

Memcached at a glance

Memcached is an open-source, in-memory key-value cache written in C, originally built by Brad Fitzpatrick at LiveJournal in 2003 and later maintained by a community of contributors. It sits in front of relational databases and remote services, holds the results of expensive reads for short windows, and hands them back on the next request without a round-trip to disk. The design is deliberately simple: a flat key space with string values up to one megabyte by default, LRU eviction inside each slab class, and either the text or binary protocol over TCP or UDP.

The server has no replication, no persistence, and no built-in sharding, which was the right call for a cache but means the client library is responsible for consistent hashing across the pool. Most of the trouble we see in production is not about the server itself but about the shape of the data: hot keys that saturate a single server, slab classes that evict unevenly because item sizes cluster oddly, serialized values that drift past the default max item size, and client timeouts that escalate into application-level outages when a node goes quiet.

Type
Distributed in-memory key-value cache
First released
2003
License
BSD 3-Clause
Maintained by
Community project with maintainers including Dormando
Typically used for
Caching database query results, session storage, API response caching, rate-limiting counters

The honest version

Memcached pros and cons

No technology is all upside. This is where Memcached earns its place and where it tends to cause trouble.

Where Memcached is strong

  • Small, focused, fastThe server does one job, holds items in memory and hands them back under a millisecond, and the codebase is small enough that operational behavior is largely predictable.
  • Battle-tested under real production loadMemcached has run at Facebook, Twitter and Wikipedia scale for years. Nothing about the server is theoretical at this point.
  • Simple protocol and wide client supportClients exist for every language you can name, and the protocol is small enough to debug with a plain telnet session when something interesting happens.
  • No persistence and no replication is a feature hereThe cache does not pretend to be a database, so there is no AOF, no snapshots and no cluster state to understand before you can turn the service on.
  • Modern builds have extstore and TLSLater releases added a flash-based extstore for large caches, TLS listeners, and a meta protocol that supports leases and CAS semantics without replacing the server.
  • Honest resource footprintA Memcached instance that is given 8GB of RAM uses roughly 8GB of RAM. There are no hidden background processes or metadata files that surprise the ops team.

Where Memcached bites

  • No native clustering or replicationConsistent hashing lives in the client, and a dead node means every key it held is a cache miss until clients notice. For many workloads that is fine; for some it is not.
  • Slab allocator causes uneven evictionItems are bucketed by size into slab classes. If your workload shifts, one class evicts while another sits half empty, and reclaim across classes is not automatic unless you enable it.
  • One-megabyte default item size capA serialized object that grows past the cap gets refused with SERVER_ERROR and the calling code often does not notice. Raising the cap has compaction costs.
  • No persistence means cold starts are costlyA restart or deploy clears the cache entirely. Applications that depend heavily on cache warmth need a priming strategy, otherwise the database gets a stampede.
  • Protocol has limited features next to RedisThere are no lists, sets, sorted sets or pub/sub. Teams reach for Memcached first and often migrate parts of the workload to Redis when they need more than get and set.
  • Observability is on youstats and stats slabs are useful and plain text. Turning them into dashboards, alerting on evictions and hit rate, and attributing load to tenants is an exercise you set up yourself.

Sound familiar?

Common Memcached problems we fix

The Memcached problems that bring people to us, and what you have once each one is fixed.

SERVER_ERROR object too large for cache

Values are exceeding the item size cap

A serialized session or API response that drifted past one megabyte as new fields were added. Writes fail silently from the application's perspective and every read becomes a database round-trip.

Values trimmed or item cap raised intentionally

slab class 12: evictions 42,118

A single slab class is evicting hot items

The working set concentrates in one size bucket that is too small for it, while other classes hold older items that nobody is reading. Hit rate drops without the memory looking full.

Slab reassignment enabled, class sizing tuned

connection timeout to 11211

Clients are timing out on a single node

A Memcached node under GC pressure from a neighboring process, or a network path that is intermittently slow. Every client sees a few percent of operations hang, which cascades into application slowness.

Node pressure removed, timeouts calibrated

hit rate 0.81 (was 0.96)

Hit rate has quietly dropped

A new feature writing short-lived keys that push older hot keys out, or a TTL that was set once and is now too aggressive for the current traffic pattern.

Working set and TTLs understood

slab reassignment disabled

Memory sits in slabs that are no longer hot

A workload that shifted between item sizes without the slab allocator being allowed to move memory. Old classes hold pages the new workload needs and would gladly use.

Slab rebalance on, reclaim observed

CLIENT_ERROR bad data chunk

Client library and server disagree on bytes

A client sending values that do not match their declared length, often after a library upgrade or a Unicode handling change. Writes fail intermittently and sometimes mid-pipeline.

Client and server agreeing on encoding

cold start, cache empty

A deploy drops the cache and the database spikes

Every restart flushes the local cache, and the application's first few seconds hammer the database with a thundering herd of cold reads.

Warm-start or request coalescing in place

connection pool exhausted

Clients run out of connections to the pool

An application that opens a new connection per worker instead of reusing a pool, or a sudden burst of traffic that consumes every available socket on the Memcached servers.

Pools sized, keepalives respected

What we do

Memcached help, from rescue to steady running

Ring us for one failing node, or let us own the cache tier end to end.

Hit rate and slab review
stats slabs analyzed against real traffic, item sizing adjusted, reclaim enabled where it helps, with numbers before and after.
Cache design
Keyspace design, TTL strategy, consistent hashing on the client, and failure handling so a dead node does not degrade the whole request path.
Memcached to Redis migration
Moving workloads that have outgrown get and set to Redis while leaving the pure cache paths where they belong, without a big-bang rewrite.
Warm starts and stampede control
Avoiding database pile-ups after deploys with request coalescing, soft TTLs or local pre-warming of hot keys.
Observability
Dashboards on hit rate, evictions, memory per class and client-side latency, with alerts tied to the thresholds that matter for your workload.
Managed Tech
Capacity reviews, patching, failover drills and someone to page when evictions climb at 2am without an obvious cause.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Memcached alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Memcached questions

Something else? Call+1 737 727 7327

Should we use Memcached or Redis?

Memcached is excellent when you need a simple, horizontally scaled, no-persistence cache in front of a database. Redis wins when you need data structures, persistence, pub/sub, scripting or cluster-wide replication. Many stacks use both: Memcached for pure read-through caching, Redis for sessions, queues and rate limiting. We will tell you honestly which one fits each part of your workload rather than pushing a single answer.

Why has our Memcached hit rate dropped?

Usually one of three things. A new feature started writing short-lived keys that push older hot items out of memory, a TTL set long ago no longer matches real traffic, or a slab class has become full while others sit nearly empty because item sizes have drifted. We read stats and stats slabs against real traffic rather than guess, and most hit-rate regressions trace back to one of those three.

How do we handle Memcached losing a node without an outage?

On the client side, consistent hashing with a modest replication factor and short connection timeouts so a dead node fails fast rather than hanging workers. On the application side, request coalescing on hot keys so a sudden flood of misses does not land on the database as a thundering herd. Replacing the node restores the cache gradually as traffic warms it.

Can you take over a Memcached deployment from a previous team?

Yes. We inventory the nodes, document which applications hash to which pool, check whether SASL is enabled and whether the listener is unexpectedly open to the world, and bring the configuration under version control before changing anything. Takeovers of inherited caches are a common piece of what we do.

How do we raise the max item size safely?

The -I flag lets you raise the cap above one megabyte, with the caveat that larger slab classes are created and more memory is reserved per class. On a busy cache you want to raise it intentionally, measure the slab distribution afterward, and often it is a sign that one particular value belongs somewhere other than a cache entirely.

How much does Memcached help cost?

We quote for free. Describe the pool, the client library and the behavior you are seeing, and we come back with a fixed price before any configuration moves.

Free estimate

Memcached giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.