Cardinality blew up without warning
A new label capturing request IDs or user IDs attached to a common metric. Memory usage climbs, GC lengthens, and PromQL queries that used to take seconds take minutes.
Cardinality back under control
Running Prometheus?
Prometheus gets stood up in an afternoon and becomes a problem a quarter later: a label somebody added to every HTTP request blew the cardinality, the WAL did not recover cleanly after a node reboot, and the alerts that were supposed to catch it were themselves scraping a dead target. We bring the series count down, the WAL back to life, and the rules back in line with the SLOs they were written for.
What it is
Prometheus is an open-source monitoring system and time-series database originally built at SoundCloud in 2012, inspired by Google's Borgmon. It joined the Cloud Native Computing Foundation in 2016 as the second hosted project after Kubernetes, and it is the de facto standard for metrics in cloud-native environments.
The model is pull based: a Prometheus server scrapes HTTP endpoints exposing metrics in a simple text format, stores the samples in a local time-series database with a write-ahead log, and exposes them through PromQL for queries, dashboards and alerts. Alertmanager handles grouping, deduplication and routing of alerts, and remote-write lets long-term storage live in Thanos, Mimir, Cortex or a managed service like Grafana Cloud.
The honest version
No technology is all upside. This is where Prometheus earns its place and where it tends to cause trouble.
Sound familiar?
The Prometheus problems that bring people to us, and what you have once each one is fixed.
A new label capturing request IDs or user IDs attached to a common metric. Memory usage climbs, GC lengthens, and PromQL queries that used to take seconds take minutes.
Cardinality back under control
A node that lost power during a write, a disk that filled up, or a kernel panic caught the WAL mid-flush. Prometheus refuses to start until the segment is dealt with.
Storage reopened, data loss minimized
An exporter now returns more series than the configured sample_limit allows, so the whole scrape is discarded. Metrics vanish from dashboards with no obvious failure.
Scrape budget sized to the workload
Remote-write buffers filled because the downstream (Thanos, Mimir, Grafana Cloud) is slow or rate limited, and Prometheus starts dropping samples it cannot ship.
Remote-write keeping up with ingest
An endpoint that takes longer than the scrape timeout to return, often because the application is exposing far too many metrics in a single response.
Scrape times inside the budget
A rule that hit the evaluation timeout, or a rule group that is thrashing because it queries too large a range. The affected alerts simply stop firing.
Rule groups green on every eval cycle
Two targets that collided on labels, usually because a relabel rule set the same instance label on both. The scrape succeeds but the samples are nonsensical.
Target identity unique and stable
Head block memory grew past the container limit, usually driven by cardinality or a long WAL replay. Restarts make it worse because replay burns more memory, not less.
Memory headroom and head size matched
What we do
Call us for one runaway label, or hand us the full metrics stack.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
Nine times out of ten it is cardinality: a label on a hot metric now carries user IDs, request IDs, or short-lived pod names, and the active series count has quietly grown by one or two orders of magnitude. The head block lives in memory, so the resident set follows. We pull the top-N offenders, drop or rewrite the labels, and watch the memory come back down over the next compaction.
Partially. The affected WAL segment can be deleted and Prometheus will start, but samples in that segment are lost. If you have remote-write to long-term storage, those samples are recoverable from there. If not, there is a gap in history. The real fix is to prevent it: a shutdown-friendly deploy, a WAL that fits on a volume with reliable fsync, and replicas that disagree if one goes bad.
For teams who want to own the stack, Thanos is the mature choice for long-term storage and global view, and Mimir is the newer Grafana Labs option that aims for simpler operation. For teams who do not want to own it, Grafana Cloud, Chronosphere, Amazon Managed Prometheus or Google Cloud Managed Prometheus take the operational burden away at a cost. The right answer depends on how much time you want to spend on the metrics plane.
Three defences stack. A relabel rule that drops obvious offenders at scrape time, a sample_limit on each scrape config so a runaway exporter fails loudly rather than quietly, and recording rules that pre-aggregate the big dashboards so queries do not fan out over every series.
Either run a second Prometheus that scrapes the first and alerts on absence, or use a dead-man switch that fires an alert to an external service every minute and pages when it stops arriving. Relying on the only Prometheus to alert on its own death is the most common mistake we see.
The quote is complimentary. Share a few PromQL queries that are slow, a screenshot of memory climbing, and a sample alert rule, and we return a bounded price before engaging further.
Free estimate
Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: