Running Prometheus?

Prometheus eating memory or losing metrics? We bring it back.

Prometheus gets stood up in an afternoon and becomes a problem a quarter later: a label somebody added to every HTTP request blew the cardinality, the WAL did not recover cleanly after a node reboot, and the alerts that were supposed to catch it were themselves scraping a dead target. We bring the series count down, the WAL back to life, and the rules back in line with the SLOs they were written for.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

prometheus.log

still failing

What it is

Prometheus at a glance

Prometheus is an open-source monitoring system and time-series database originally built at SoundCloud in 2012, inspired by Google's Borgmon. It joined the Cloud Native Computing Foundation in 2016 as the second hosted project after Kubernetes, and it is the de facto standard for metrics in cloud-native environments.

The model is pull based: a Prometheus server scrapes HTTP endpoints exposing metrics in a simple text format, stores the samples in a local time-series database with a write-ahead log, and exposes them through PromQL for queries, dashboards and alerts. Alertmanager handles grouping, deduplication and routing of alerts, and remote-write lets long-term storage live in Thanos, Mimir, Cortex or a managed service like Grafana Cloud.

Type
Metrics collection, time-series database, and alerting engine
First released
2015 publicly; development began at SoundCloud in 2012
License
Apache 2.0, open source
Maintained by
Prometheus Authors under the CNCF
Typically used for
Service and infrastructure metrics, SLO monitoring, alert routing, Kubernetes observability

The honest version

Prometheus pros and cons

No technology is all upside. This is where Prometheus earns its place and where it tends to cause trouble.

Where Prometheus is strong

  • The de facto cloud-native metrics standardEvery Kubernetes ingress, service mesh, database and runtime exposes Prometheus-format metrics. There is almost nothing in the CNCF ecosystem it does not already see.
  • PromQL is powerful for ad hoc analysisRate, histogram_quantile, aggregations and subqueries let an engineer chase a latency spike or compute an SLO burn rate without a bespoke dashboard.
  • Pull model simplifies securityTargets do not need to know about the server, and firewall rules only have to allow outbound Prometheus to reach each service, which plays well with private networks.
  • Service discovery is first-classKubernetes, Consul, EC2, GCE and file-based service discovery bring targets in without configuration drift between what is running and what is being scraped.
  • Alertmanager actually models real alert routingGrouping, inhibition, silences and receiver trees reflect how on-call rotations really work rather than firing a thousand messages for a single outage.
  • Remote-write ecosystem for long-term storageThanos, Cortex, Mimir and managed vendors all plug in via remote-write, so local retention stays small and long-term queries have somewhere to live.

Where Prometheus bites

  • High cardinality is the silent killerA label with user IDs, request IDs, or Kubernetes pod names will blow memory and query performance long before an engineer notices. Recovery is painful once it is in the WAL.
  • Single-node storage is deliberately simpleLocal disk, no replication, no clustering. Any HA or long-term retention story requires Thanos, Cortex, Mimir or a managed service, which is its own operations bill.
  • WAL corruption after an ungraceful shutdownA node that lost power mid-write can leave the write-ahead log in a state that refuses to replay. Recovery involves deleting blocks and accepting the data loss.
  • Alerting on the thing that is brokenIf Prometheus is down, so are the alerts that would have told you. A second Prometheus, or a dead-man switch to an external service, is needed and often skipped.
  • Query performance degrades with series countA dashboard that was snappy at 500k series becomes a tar pit at 5M. Recording rules help but need discipline to maintain.
  • Operational burden is non-trivialConfig reload loops, rules linting, scrape timeouts, Alertmanager routing trees and remote-write queues each have their own failure modes. None of them are hard, and all of them add up.

Sound familiar?

Common Prometheus problems we fix

The Prometheus problems that bring people to us, and what you have once each one is fixed.

label values exceed max cardinality

Cardinality blew up without warning

A new label capturing request IDs or user IDs attached to a common metric. Memory usage climbs, GC lengthens, and PromQL queries that used to take seconds take minutes.

Cardinality back under control

WAL corruption detected

Storage will not open after a restart

A node that lost power during a write, a disk that filled up, or a kernel panic caught the WAL mid-flush. Prometheus refuses to start until the segment is dealt with.

Storage reopened, data loss minimized

sample limit of 50000 exceeded

Scrape targets drop samples silently

An exporter now returns more series than the configured sample_limit allows, so the whole scrape is discarded. Metrics vanish from dashboards with no obvious failure.

Scrape budget sized to the workload

remote_write queue full

Long-term storage falls behind

Remote-write buffers filled because the downstream (Thanos, Mimir, Grafana Cloud) is slow or rate limited, and Prometheus starts dropping samples it cannot ship.

Remote-write keeping up with ingest

context deadline exceeded on scrape

Targets time out during scrape

An endpoint that takes longer than the scrape timeout to return, often because the application is exposing far too many metrics in a single response.

Scrape times inside the budget

rule_group failed to evaluate

Recording and alert rules stop running

A rule that hit the evaluation timeout, or a rule group that is thrashing because it queries too large a range. The affected alerts simply stop firing.

Rule groups green on every eval cycle

ingesting samples with different value but same timestamp

Duplicate series slip in

Two targets that collided on labels, usually because a relabel rule set the same instance label on both. The scrape succeeds but the samples are nonsensical.

Target identity unique and stable

Memory usage approaching limit

The Prometheus pod gets OOM killed

Head block memory grew past the container limit, usually driven by cardinality or a long WAL replay. Restarts make it worse because replay burns more memory, not less.

Memory headroom and head size matched

What we do

Prometheus help, from one noisy alert to the whole stack

Call us for one runaway label, or hand us the full metrics stack.

Rescue
Prometheus OOMing, WAL refusing to replay, or every alert firing at once. We stabilize the server and triage the series afterwards.
Cardinality audits
Top-N series by metric and label, a plan to drop or relabel the worst offenders, and recording rules to make the dashboards snappy again.
Alert and SLO design
Alertmanager routing trees that match the real on-call, multi-window burn rate SLOs, and runbooks attached to every rule so the person paged knows what to do.
Long-term storage
Thanos, Mimir or a managed backend wired in through remote-write, with retention and query performance tested on real data rather than demo volumes.
Migrations and upgrades
Prometheus version upgrades without data loss, moving from statsd or Graphite, and swapping out a managed vendor without losing alert continuity.
Managed Tech
Ongoing capacity and cardinality watch, rule review when something starts flapping, and a team to call when a scrape endpoint rots at midnight.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Prometheus alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Prometheus questions

Something else? Call+1 737 727 7327

Why is my Prometheus server running out of memory?

Nine times out of ten it is cardinality: a label on a hot metric now carries user IDs, request IDs, or short-lived pod names, and the active series count has quietly grown by one or two orders of magnitude. The head block lives in memory, so the resident set follows. We pull the top-N offenders, drop or rewrite the labels, and watch the memory come back down over the next compaction.

Can I recover from a Prometheus WAL corruption?

Partially. The affected WAL segment can be deleted and Prometheus will start, but samples in that segment are lost. If you have remote-write to long-term storage, those samples are recoverable from there. If not, there is a gap in history. The real fix is to prevent it: a shutdown-friendly deploy, a WAL that fits on a volume with reliable fsync, and replicas that disagree if one goes bad.

Should we use Thanos, Mimir, or a managed Prometheus service?

For teams who want to own the stack, Thanos is the mature choice for long-term storage and global view, and Mimir is the newer Grafana Labs option that aims for simpler operation. For teams who do not want to own it, Grafana Cloud, Chronosphere, Amazon Managed Prometheus or Google Cloud Managed Prometheus take the operational burden away at a cost. The right answer depends on how much time you want to spend on the metrics plane.

How do I stop high-cardinality labels before they hurt?

Three defences stack. A relabel rule that drops obvious offenders at scrape time, a sample_limit on each scrape config so a runaway exporter fails loudly rather than quietly, and recording rules that pre-aggregate the big dashboards so queries do not fan out over every series.

How do I alert on Prometheus itself being down?

Either run a second Prometheus that scrapes the first and alerts on absence, or use a dead-man switch that fires an alert to an external service every minute and pages when it stops arriving. Relying on the only Prometheus to alert on its own death is the most common mistake we see.

How much does Prometheus help cost?

The quote is complimentary. Share a few PromQL queries that are slow, a screenshot of memory climbing, and a sample alert rule, and we return a bounded price before engaging further.

Free estimate

Prometheus giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.