Running ScyllaDB?

ScyllaDB repairs stalling or shards misrouted? We unstick it.

ScyllaDB rewards operators who understand its shard-per-core model and punishes the ones who treat it like a drop-in Cassandra. We sort out the drivers, the repair jobs, the compaction storms and the token-ring quirks until the cluster behaves the way the benchmarks promised.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

scylla-nodetool.log

still failing

What it is

ScyllaDB at a glance

ScyllaDB is a wide-column NoSQL database written in C++ that reimplements the Apache Cassandra protocol on top of the Seastar framework. It uses a shard-per-core architecture and asynchronous I/O to push far more work through the same hardware than a JVM-based Cassandra cluster typically manages.

The company behind it, ScyllaDB Inc., offers an open-source edition, a self-managed enterprise build, and ScyllaDB Cloud as a hosted service. Common deployments include session storage, time-series workloads, ad-tech serving layers and IoT ingestion where predictable low-latency reads under heavy write pressure matter most.

Type
Wide-column NoSQL database, Cassandra-compatible
First released
2015
License
AGPL-3.0 open source core, proprietary enterprise edition
Maintained by
ScyllaDB Inc.
Typically used for
High-throughput key-value and time-series serving, session stores, telemetry pipelines

The honest version

ScyllaDB pros and cons

No technology is all upside. This is where ScyllaDB earns its place and where it tends to cause trouble.

Where ScyllaDB is strong

  • Far more work per nodeThe shard-per-core design pins each CPU to its own slice of memory and SSTable set, which usually means a fraction of the nodes a Cassandra cluster of the same shape would need.
  • Predictable tail latencyP99 numbers stay flat under pressure because garbage collection pauses and locking contention from the JVM world simply do not exist in the C++ engine.
  • Cassandra driver compatibilityExisting CQL applications connect without code changes, which makes trialling ScyllaDB behind an existing workload a short exercise rather than a rewrite.
  • Built-in workload prioritizationService levels let you reserve shares of a shard for OLTP traffic so a runaway analytics query cannot starve the user-facing path.
  • Honest operational toolingThe Scylla Monitoring Stack, Scylla Manager and nodetool extensions surface the sort of internals operators actually need instead of hiding them.
  • Hosted option availableScyllaDB Cloud absorbs the upgrade work, the repair scheduling and the capacity planning for teams that do not want to run the ring themselves.

Where ScyllaDB bites

  • Hiring Scylla operators is hardThe pool of engineers who have actually debugged a stuck repair or a shard-skew problem is small, so inherited clusters often sit with nobody willing to touch them.
  • Shard-awareness is unforgiving behind NATContainers, service meshes and cloud load balancers routinely rewrite source ports in ways that confuse the shard-aware driver, producing warnings that take real time to trace.
  • Large partitions bite hardA data model that worked at low volume turns into a single shard doing all the work the moment one key accumulates too many clustering rows, and the symptom is latency not an error.
  • Repair and compaction compete for diskStreaming repairs, major compactions and tombstone cleanup all want I/O at the same moment, and without careful scheduling they slow production writes to a crawl.
  • AGPL licensing on the open-source coreThe strong copyleft terms rule it out for a lot of SaaS products, pushing teams toward the enterprise edition or the hosted service sooner than they planned.
  • Upgrades across major versions need a rehearsalProtocol and SSTable format changes between majors have caught teams out, so skipping the staging run is the sort of mistake that produces a weekend incident.

Sound familiar?

Common ScyllaDB problems we fix

The ScyllaDB problems that bring people to us, and what you have once each one is fixed.

put_row_diff: Repair follower failed

Repair sessions fail halfway through

A streaming repair that gets partway across the ring and dies on a checksum mismatch or a connection reset, leaving replicas in an unclear state and nodetool reporting partial success.

Repair sessions finishing cleanly

expected shard 0

The shard-aware driver keeps misrouting

Connections landing on the wrong shard because a NAT boundary or service mesh is rewriting source ports, costing you the latency advantage the driver was meant to deliver.

Driver routing honest again

large partition detected: 384 MB

A hot partition is dragging one shard down

A data model where one partition key collects too many rows, leaving a single core doing all the work while the others sit idle and the P99 drifts upward.

Partitions rebalanced across the ring

compaction failed: std::bad_alloc

Compaction falls behind under load

SSTables accumulating faster than compaction can merge them, which produces read amplification, memory pressure and eventually out-of-memory errors on the shards with the hottest tables.

Compaction caught up, headroom restored

token ring change detected

Topology changes are painful

Replacing a dead node, growing the cluster or swapping instance types ends up streaming far more data than expected and taking far longer than the maintenance window allowed.

Topology change rehearsed and completed

gossip: node 10.0.0.9 marked DOWN

Nodes flap in and out of the ring

Short network hiccups, GC pauses on a sidecar or a saturated CPU sending gossip messages late, with the cluster marking otherwise healthy nodes as down and triggering hints that never clear.

Gossip settled, flap stopped

AGPL incompatible with product license

The license is blocking a release

Legal review flagging the AGPL-3.0 core on an edge of your product that cannot ship under copyleft terms, which is a surprisingly late discovery for a lot of teams.

Path forward, enterprise or hosted, decided

What we do

ScyllaDB help, from rescue to capacity planning

Reach out for one stuck repair session, or hand us the full ring and the drivers in front of it.

Rescue
A repair that will not finish, compaction falling behind, or a replacement node that will not join. We stabilize the ring first and explain the cause second.
Data modeling review
Partition keys, clustering columns and secondary indexes checked against how the application actually reads and writes, with the hot partitions renamed before they bite.
Driver and client tuning
Shard-aware connections working through NAT, retry policies that respect service-level targets, and load balancing that stops starving any one coordinator.
Capacity and cost review
Instance types, replication factor, compaction strategies and backup retention examined against the real workload rather than the original sizing guess.
Migration from Cassandra
Running Scylla and Cassandra side by side, cutting reads over first, then writes, with sstableloader or nodetool refresh handling the historical data.
Managed Tech
Repair schedules kept honest, upgrades rehearsed on a replica ring, and an operator on call when gossip starts misbehaving at 3am.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running ScyllaDB alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

ScyllaDB questions

Something else? Call+1 737 727 7327

Should we move from Cassandra to ScyllaDB?

Often yes, if the workload is the kind Cassandra struggles with: steady high write volume, latency-sensitive reads and node counts you would rather not keep growing. The CQL compatibility means the application change is small; the operational change is the bigger lift, and that is where a rehearsed migration pays for itself.

Why are our repair sessions failing?

Most commonly a mix of flaky network paths between racks, oversized partitions that break the streaming budget, and repair parallelism set too high for the hardware. Scylla Manager with range-splitting usually fixes it, but the underlying cause still needs looking at or the next repair fails the same way.

How do we handle a hot partition?

You find it first using nodetool toppartitions or the large-partition warnings in the logs, then redesign the partition key so the hot access pattern spreads across multiple keys rather than landing on one shard. Sometimes a bucketing scheme on a timestamp is enough; sometimes the schema needs a wider rework.

Can you run Scylla on Kubernetes?

Yes, using the Scylla Operator with local SSDs and dedicated nodes rather than generic cloud disks. The operator handles bootstrap, upgrades and scaling; the caveat is that stateful workloads on Kubernetes still require serious attention to storage and taints, not a copy-paste manifest.

How much does ScyllaDB help cost?

Quotes are no charge. Share the ring size, the workload pattern, and the error you are chasing, and we respond with a number before any work starts.

Free estimate

ScyllaDB giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.