Running Cassandra?

Cassandra ring under pressure? We clear the backlog.

Cassandra rewards good schema decisions made years ago and punishes everyone else. Most of the pages we get are about tombstone warnings swamping a read path, a repair that will not finish, a compaction backlog that keeps growing, or an UnavailableException arriving every time traffic peaks. We work the ring one node at a time, fix the real cause, and leave a runbook so the next incident is not the same one.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

cassandra-system.log

still failing

What it is

Apache Cassandra at a glance

Apache Cassandra is a wide-column, distributed database originally built at Facebook and released as open source in 2008. It gives you linear horizontal scaling across a ring of nodes, tunable consistency on every query, and no single point of failure. The query language, CQL, looks reassuringly like SQL and hides a storage engine that behaves nothing like a relational database.

It is the right tool for workloads that have very high write throughput, geographically distributed reads and writes, and a schema you can design around the partition key from day one. It is the wrong tool for ad hoc analytics, for small datasets, or for teams that expect to add new query patterns on a whim. Alternatives in the same neighbourhood include ScyllaDB, which is API-compatible, and the managed Astra DB from DataStax.

Type
Distributed wide-column NoSQL database
First released
2008 open-source release
License
Apache License 2.0
Maintained by
The Apache Software Foundation
Typically used for
High-throughput time-series data, event stores, IoT telemetry, user-activity logs, large-scale catalogue services

The honest version

Apache Cassandra pros and cons

No technology is all upside. This is where Apache Cassandra earns its place and where it tends to cause trouble.

Where Apache Cassandra is strong

  • Writes scale linearly with the ringAdding nodes gives you more write capacity in a very predictable way, which is why teams with heavy write workloads reach for Cassandra over a sharded relational database.
  • No single point of failureEvery node is a peer. Losing one, or a whole rack, or a data centre, degrades capacity but does not take the cluster offline when replication is set up correctly.
  • Tunable consistency per operationChoosing between ONE, QUORUM and LOCAL_QUORUM per statement lets you trade latency against durability where it actually matters, rather than at the database level.
  • Multi-region is a first-class featureDatacenter awareness is built into the replication strategy and the driver, which makes active-active across regions feasible without bolt-on tooling.
  • CQL keeps the learning curve manageableThe query language and the driver model let a team ship a first application without first mastering the storage engine, provided someone later tightens the schema for real workloads.

Where Apache Cassandra bites

  • Schema design mistakes are expensive to undoA partition key that produces hot partitions or allows unbounded growth is very hard to fix at scale, and the symptoms often appear only once production data is in.
  • Tombstones are a recurring source of painDeletes and TTL expiry create tombstones that have to be scanned on reads until compaction removes them, and workloads that delete often will see read-path warnings and eventual failures.
  • Repairs are not optional and not quickRunning regular full or incremental repairs is required to keep replicas consistent, and on large data sets a repair can take hours and can fail in ways that need operator intervention.
  • JVM tuning is part of the jobGarbage collection pauses, heap sizing and the compaction strategy interact. A default config that was fine at launch starts throwing GCInspector warnings and timeouts as data grows.
  • Operational overhead is realCompaction, repair, node joins and leaves, snapshot management and gossip debugging all need attention. Cassandra does not run itself even on a healthy cluster.
  • Hiring is harder than for mainstream databasesEngineers who genuinely understand Cassandra's storage model and operations are a smaller pool than Postgres or MongoDB people, and the cost shows up in production incidents.

Sound familiar?

Common Apache Cassandra problems we fix

The Apache Cassandra problems that bring people to us, and what you have once each one is fixed.

UnavailableException: Cannot achieve consistency level QUORUM

Writes start failing during a node outage

Replication factor and consistency level that do not survive losing a node, or a datacenter configuration that treats one rack as authoritative when it should not.

Consistency topology matched to real loss scenarios

Read 42013 tombstone cells

A read path is drowning in tombstones

A table that deletes rows or uses short TTLs as part of its design, where compaction cannot keep up and queries scan huge tombstone ranges to return a handful of live cells.

Schema, TTL and compaction aligned

RepairJob failed Session completed with error

Repair sessions will not complete

A long-running repair that times out on large tables, a streaming session interrupted by a flapping node, or a schema disagreement between nodes blocking the merkle tree comparison.

Repairs running on a schedule that works

Pending tasks in CompactionExecutor: 142

Compaction is falling behind

A workload that writes faster than compaction can clean up, or a compaction strategy that does not match the access pattern, so sstable count climbs and reads get slower.

Compaction strategy suited to the workload

GCInspector ParNew GC in 4211ms

GC pauses break latency SLOs

Heap sized at a default that no longer fits the working set, or a GC algorithm that is not the right choice for the current data volume, producing multi-second stop-the-world pauses.

JVM tuned for the real workload

OverloadedException pending requests

A hot partition is throwing errors under load

A partition key that funnels disproportionate traffic to a single node, so the ring is otherwise idle while one node rejects requests with overload errors.

Hot partitions broken up

SSTable corrupt negative value length

An SSTable is corrupt

Hardware failure, a kill during a flush, or a disk that returned bad data silently, leaving one or more sstables unreadable and the node refusing to serve the range.

Node recovered, data repaired from replicas

endpoint has no owners for the ranges

Repair reports ranges with no owner

A cluster whose tokens got out of sync after a bootstrap failure or a manual nodetool removenode, so parts of the ring are not covered by any replica at all.

Ownership restored, ring clean

What we do

Cassandra help, from rescue to steady state

Call us for one failing repair, or hand us the whole ring.

Rescue
A cluster throwing UnavailableException under peak load, a repair that will not finish, or a node that will not rejoin. We stabilise the ring first and explain what tripped it after.
Schema and data modelling
Reviewing partition keys, clustering keys and TTLs against your actual access patterns, catching hot partitions and unbounded growth before they become incidents.
Compaction and repair
Compaction strategy chosen deliberately, incremental or full repair scheduled, and the operational tooling to catch a failed session before it blocks a bigger window.
JVM and node tuning
Heap sizing, GC algorithm, read-ahead and compression settings tuned against real workload traces, not inherited from a three-year-old blog post.
Multi-region and upgrades
Multi-datacenter topology, LOCAL_QUORUM strategies, and version upgrades from 3.x or 4.x onto a supported line without a long outage.
Managed Tech
Repair orchestration, compaction monitoring, snapshot lifecycle, upgrade planning, and an engineer available when a node leaves the ring at 3 a.m.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Apache Cassandra alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Apache Cassandra questions

Something else? Call+1 737 727 7327

Should we actually be using Cassandra for this workload?

Honestly, often not. Cassandra earns its keep for massive write throughput, true multi-region active-active, or data volumes a single Postgres cannot hold. For most line-of-business workloads a tuned relational database is cheaper, easier to hire for, and less operationally demanding. We will tell you plainly when the cluster exists mostly because of a decision made under different assumptions.

Why do our reads slow down over time?

Three usual causes. Tombstones accumulating on tables with high delete or TTL rates, so reads scan huge ranges of dead cells. Compaction falling behind, leaving too many sstables per query. And a working set that has outgrown the current heap, so GC pauses show up as read latency. Each of them has a specific fix, but they are often intertwined on a cluster that has been running for years.

Is ScyllaDB a drop-in replacement for Cassandra?

Close but not identical. Scylla speaks the same CQL and is API-compatible with the drivers, with a different storage engine that often delivers better single-node throughput. Migrations are usually feasible but need a careful plan around drivers, compaction, and operational tooling, which is not always the same between the two.

How do we run repairs without impacting production?

Incremental repairs, scheduled with a tool like Reaper or Cassandra's built-in scheduler, chunked across token ranges and time windows that avoid your peak traffic. Full repairs on huge tables are reserved for exceptional cases. The goal is that no single repair operation lasts long enough to collide with the next scheduled one.

Can we move from Cassandra to a managed service like Astra DB?

Usually, yes. Astra DB is DataStax's managed Cassandra and supports the same CQL, with migration tooling for the schema and the data. The economics depend on your scale and traffic shape. For some teams a managed service is a strict operational improvement; for others the egress and seat costs push them to keep running self-managed.

How much does Apache Cassandra help cost?

Estimates are complimentary. Describe the ring size, version, and the specific errors or latency you are seeing, and you get a quote back before any node is touched.

Free estimate

Cassandra giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.