Running CockroachDB?

CockroachDB aborting transactions or dragging at scale? We sort it out.

CockroachDB promises Postgres compatibility with the write anywhere, read anywhere story of a distributed system, and it mostly delivers. What it also delivers, once you push past a demo, is retry logic you must handle, latency patterns that do not look like a single node, and clock skew warnings that stop the cluster accepting writes. We wire the retries, pick the locality and tune the ranges until the cluster behaves.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

cockroach-stderr.log

still broken

What it is

CockroachDB at a glance

CockroachDB is a distributed SQL database built by Cockroach Labs, founded in 2015 by former Google engineers. It speaks the Postgres wire protocol, which means most clients and ORMs work without modification, and it stores data in replicated ranges across nodes using the Raft consensus algorithm.

The database targets workloads that need horizontal scale, survivability across regions and strong transactional guarantees at the same time. It is available as self hosted open source, a managed CockroachDB Cloud with dedicated or serverless tiers, and an Enterprise license for self hosted deployments that need multi region features. Teams usually pick it when a single Postgres primary is no longer surviving the write load or when regulatory requirements demand data stay in specific regions.

Type
Distributed SQL database, Postgres wire compatible
First released
2015 beta, 1.0 in 2017
License
Apache 2.0 core, CCL for enterprise features, managed CockroachDB Cloud
Maintained by
Cockroach Labs, Inc.
Typically used for
Multi region OLTP, financial systems, regulated workloads, anything that outgrew a single Postgres primary

The honest version

CockroachDB pros and cons

No technology is all upside. This is where CockroachDB earns its place and where it tends to cause trouble.

Where CockroachDB is strong

  • Postgres wire compatibleExisting drivers, pgx, psycopg and most ORMs connect without changes, which keeps migration cost down compared with a brand new SQL dialect.
  • Survives region lossWith three or more regions configured, the cluster keeps serving reads and writes when one goes dark, which is a hard property to get out of a single primary setup.
  • Scales writes horizontallyRanges split and rebalance across nodes automatically, so adding capacity is a matter of adding nodes rather than running a shard migration.
  • Serializable isolation by defaultNo read committed anomalies to debug, which closes off a whole class of correctness bugs that plague higher concurrency Postgres applications.
  • Managed Cloud offeringCockroachDB Cloud takes cluster operations, upgrades and backup off your plate on dedicated or serverless tiers, which suits teams without a database specialist.
  • Real multi region primitivesTable and row level locality clauses let you keep European rows in Europe and American rows in America, which matters for both latency and regulation.

Where CockroachDB bites

  • Transaction retries are on the applicationSerializable isolation means the database legitimately aborts conflicting transactions, and the client has to retry them. Code that treats a retry error as a hard failure will lose writes under load.
  • Latency is not single node latencyA write is only committed once a majority of replicas acknowledge it, so cross region p99 latency is measured in tens of milliseconds at best. Assumptions carried over from a local Postgres will feel slow.
  • Clock synchronisation really mattersNodes compare commit timestamps across the cluster, and a drifting clock on one node can take it out of the quorum. NTP has to be set up properly, and chrony is not a nice to have.
  • Some SQL features are missing or differentStored procedures, listen notify, extensions like PostGIS at parity, and triggers either arrived late or still differ from Postgres. Code that relied on them needs a plan.
  • Enterprise and Cloud get expensive fastMulti region, change data capture, backup to cloud storage and more are gated behind commercial licensing, and the per node pricing climbs quickly for real clusters.
  • Operational depth is substantialRanges, leases, zone configs, backup schedules and upgrade orchestration are all part of running a cluster, and the knowledge curve is steeper than a single Postgres instance.

Sound familiar?

Common CockroachDB problems we fix

The CockroachDB problems that bring people to us, and what you have once each one is fixed.

TransactionRetryWithProtoRefreshError

Transactions keep aborting under load

Serializable conflicts legitimately aborting transactions that were not written with retry handling, with the client treating each abort as a permanent error.

Retry loop implemented, aborts handled

clock synchronization error

A node dropped out over clock skew

NTP not running, a virtualisation host with a drifting clock, or chrony misconfigured, leaving a node more than the configured offset away from its peers.

Clocks aligned, nodes rejoined

SQL txn latency p99: 840ms (was 60ms)

Latency climbed after a region was added

Writes that used to stay in one zone now have to round trip to a majority of regions, and nobody set regional by row on the hot tables.

Localities set, latency back down

result is ambiguous: context deadline exceeded

Clients see ambiguous commit results

A commit that timed out before CockroachDB could report success or failure, which is correct behaviour but confusing to clients without idempotent logic.

Writes idempotent, timeouts handled

WriteTooOldError

Long running transactions keep losing

An analytical query inside a read write transaction holding a snapshot long enough that newer writes force it to restart, which it does over and over.

Read only path or AS OF SYSTEM TIME

range has not received a snapshot in over 1h

Under replicated ranges after a node event

A node left the cluster and the replacement is not catching up, often because of disk pressure, a network hiccup or a misconfigured locality.

Ranges replicated, cluster green

node is draining

Rolling upgrades hitting application errors

A draining node is rejecting new connections while clients are still pointed at it because the load balancer did not catch the health change.

Drain sequence and LB in step

previous team gone

Inherited CockroachDB cluster

A cluster set up by a prior team, with zone configs that nobody remembers choosing, backups that may not have been tested, and a licence that is about to lapse.

Cluster documented and owned by you

What we do

CockroachDB help, from rescue to multi region

Debug one failing transaction, or let us take the whole cluster on.

Rescue
A cluster with under replicated ranges, a node that will not rejoin, or latency that has crept past what the application can tolerate. We get the cluster healthy and explain how it got there.
Retry logic in your app
Serializable retries implemented in whatever language your service is written in, with jitter, backoff and clear logging so a conflict is not mistaken for a failure.
Multi region setup
Zone configs, regional by row, regional by table and global tables chosen for your traffic, with p99 latency measured region by region.
Postgres migration
Moving a maxed out Postgres primary onto CockroachDB, including the schema quirks, the queries that do not translate cleanly, and the client retry story.
Cloud or self hosted
A frank cost comparison between CockroachDB Cloud and running it yourself at your scale, with the operational hours taken into account rather than hand waved.
Managed Tech
Backups verified, upgrades planned, cluster monitoring in place and a human on call who has read your schema when something strange shows up in the logs.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running CockroachDB alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

CockroachDB questions

Something else? Call+1 737 727 7327

Why do we keep seeing TransactionRetryWithProtoRefreshError?

This is the database doing its job. Under serializable isolation, conflicting concurrent transactions get one winner and one retry, and clients are expected to re run the transaction on that error. Most applications hitting it have no retry wrapper. We add one with backoff and jitter, and the errors stop being user visible.

Our CockroachDB cluster is slower than our old Postgres. Why?

A write in CockroachDB is only acknowledged when a majority of replicas have it. On a single region cluster that is a few milliseconds extra. On a multi region setup it can be fifty milliseconds or more per round trip. The fix is choosing locality per table or row so that writes stay close to their users, and reads come from a nearby replica.

How do we migrate from Postgres to CockroachDB?

Schema mostly transfers, but extensions like PostGIS at older versions, custom types, triggers and stored procedures need review. We dual write during the cutover, verify row counts and checksums, teach the application retry handling before the switch, and hold the Postgres instance as a fallback until the new cluster has run clean for a week.

Do we need CockroachDB or would Postgres suffice?

If a single Postgres primary comfortably handles your writes and you do not have hard region residency requirements, stay there. CockroachDB earns its complexity when you either need to survive a region loss without downtime, need to keep data in specific regions for compliance, or have genuinely outgrown vertical scale. We will say so if your case does not clear that bar.

What causes clock synchronization errors in the cluster?

NTP not running, chrony misconfigured, the virtualisation host clock drifting, or a VM pausing long enough that its clock falls behind. The cluster compares commit timestamps across nodes, and a node too far out of step is removed until its clock is back in range. We set up proper time synchronisation and monitoring so the signal is visible before it causes an outage.

How much does CockroachDB help cost?

Estimates are not billed. Share your cluster topology and an example of the behaviour you are seeing, and you get a price in writing before we start.

Free estimate

CockroachDB giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.