Running InfluxDB?

InfluxDB cardinality or retention misbehaving? We clear the backlog.

A metrics bucket has crossed into the millions of unique series, Flux tasks are running past their schedule and overlapping each other, and a retention policy set years ago is now dropping data the grafana dashboards still try to query. We open the metadata on the instance, measure which tags are generating the cardinality, and bring the write and query paths back to numbers you can plan around.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

influxd.log

still broken

What it is

InfluxDB at a glance

InfluxDB is a purpose-built time-series database, designed around timestamped points with tags and fields, built by InfluxData in 2013. It stores measurements in TSM files on disk, indexes tag values in a separate structure, and offers two query languages: InfluxQL from the 1.x line and Flux from the 2.x line. The engine is tuned for append-heavy write patterns, high-cardinality metric names are allowed but high-cardinality tag values are the single most common cause of operational trouble.

The product line has moved more than most databases of its size. Version 1.x used InfluxQL and retention policies, 2.x unified storage and introduced Flux and buckets, 3.x (also marketed as InfluxDB IOx) was rewritten in Rust on Apache Arrow and Parquet, and the Cloud offering mixes versions depending on the plan. In production, the problems are reliably about cardinality, retention policies that do not match the dashboards, tasks that overlap because the schedule drifted, and Telegraf outputs that buffer and drop silently when the instance pushes back.

Type
Time-series database
First released
2013
License
MIT (OSS), proprietary InfluxDB Cloud and Enterprise
Maintained by
InfluxData, Inc.
Typically used for
Infrastructure monitoring, application metrics, IoT telemetry, industrial sensor data, financial time-series

The honest version

InfluxDB pros and cons

No technology is all upside. This is where InfluxDB earns its place and where it tends to cause trouble.

Where InfluxDB is strong

  • Built around the write pattern it actually servesAppend-only, timestamp-ordered writes compress well into TSM files, and the engine does not pretend to be a general-purpose database that happens to handle time.
  • A clear ingest path through TelegrafHundreds of input plugins, a stable line protocol, and consistent authentication mean getting metrics into the instance rarely needs a custom agent.
  • Downsampling and continuous queries are a first-class featureTasks or continuous queries aggregate high-resolution data into coarser buckets on a schedule, so dashboards over long windows do not reread raw points.
  • Flux is powerful for shaped dataJoin, pivot and windowing operations on time-series produce the kind of graphs that would take a procedural script in a relational database.
  • Retention policies are explicitYou declare how long each bucket keeps data and InfluxDB deletes shards when they age out, which keeps disk usage honest.
  • A healthy ecosystem around GrafanaGrafana, Chronograf and Kapacitor all speak the InfluxDB protocols fluently, so visualization and alerting are a short config rather than a custom integration.

Where InfluxDB bites

  • Cardinality is the quiet killerTags with unbounded values, such as request IDs or user emails, inflate the series index until writes slow down and queries time out. By the time you notice, dropping them is painful.
  • The major version story is confusing1.x uses InfluxQL and retention policies, 2.x switched to Flux and buckets, 3.x rewrote the engine in Rust. Upgrading across those lines is a migration project rather than a patch.
  • Flux has a steep learning curveThe functional, pipe-based syntax is clean once you internalize it and frustrating until then. Teams who only ever wrote InfluxQL often spend weeks getting productive on 2.x.
  • OSS clustering ended with 1.xIf you want horizontally scaled InfluxDB you go to Cloud or Enterprise, which changes the pricing conversation.
  • Backups are version-dependentIncremental backups and point-in-time recovery differ between 1.x, 2.x and 3.x, and getting them right is one of those things that only matters until the day it is the only thing that matters.
  • Write rejections ripple through TelegrafWhen the instance returns 429 or 503, Telegraf buffers up to its configured size and silently drops the overflow. Monitoring systems lose data without an obvious alarm unless you built one yourself.

Sound familiar?

Common InfluxDB problems we fix

The InfluxDB problems that bring people to us, and what you have once each one is fixed.

series cardinality 1,842,311 approaches limit

Cardinality has run away

A tag that was supposed to hold a short enumerated list is now holding request IDs or UUIDs, and the series index has exploded. The database is still accepting writes, just slower every day.

High-cardinality tags moved to fields

field type conflict

A measurement has a field type conflict

A metric that started as an integer began arriving as a float or a string from a new client, and the TSM engine refuses the write. Points are silently dropped at ingest.

Field types consistent, writes accepted

retention policy autogen set to 30d

Dashboards query data the retention policy already dropped

A grafana panel asking for 90 days of history against a bucket whose retention is 30 days. The query returns nothing for the earlier window and no one notices until an investigation needs it.

Retention matched to the dashboard range

task 'downsample_5m' timeout exceeded

A Flux task is overrunning its schedule

A downsampling task scheduled every five minutes that now takes six minutes to run, so the next run queues behind it, memory climbs, and the cluster quietly falls behind.

Task narrowed, schedule aligned

429 Too Many Requests on Telegraf output

Telegraf is being rate-limited into dropping data

Write bursts exceed the instance's accepted rate, the buffer in Telegraf fills, overflow is dropped without an alert, and Grafana shows gaps whose cause is invisible.

Write rate smoothed, buffer monitored

cache snapshot failed: no space left

The disk has filled up

A combination of large WAL files, TSM compaction holding on to old files longer than expected, and a retention policy that does not drop fast enough. Writes stop entirely until space comes back.

Storage sized to real growth

token missing write permission

Auth tokens have the wrong scope

A token that works in the UI but fails on Telegraf because the required write bucket is not in its scope, or because it was created against the wrong organization in the 2.x model.

Tokens scoped and documented

Flux: type error

Flux queries break after a version bump

A Flux script that worked on an older server fails after upgrade because the type system tightened or a function signature changed. The error says what it saw, not what you meant.

Flux brought up to the running version

What we do

InfluxDB help, from rescue to day-to-day

Ring us for one failing task, or hand us the metrics stack whole.

Rescue
Instances with write errors, overlapping tasks, or a disk that just filled. We stabilize ingest first and write the incident up afterward.
Cardinality review
Tag design audited against the real queries, offenders found, and a plan to move them to fields or a different storage pattern without losing history.
Retention and downsampling
Retention policies, task schedules and bucket design set against what the dashboards actually ask for, so disk usage and query latency both come down.
Version upgrades and migrations
Moving between 1.x and 2.x, or planning the jump to 3.x, with parallel instances and a shadow query stage so Flux and InfluxQL both keep working.
Telegraf and ingest
Buffer sizes, flush intervals, outputs and backpressure handled so that write rejections become visible alerts rather than silent gaps.
Managed Tech
Cardinality alarms, task health, backup verification and on-call engineers for the night the TSM compaction decides to misbehave.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running InfluxDB alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

InfluxDB questions

Something else? Call+1 737 727 7327

Why has our InfluxDB slowed down even though hardware has not changed?

It is almost always series cardinality. A tag that was a small enumeration has started receiving user IDs, request IDs or free-form labels, and the index has grown into the millions. Writes slow because the engine has to update more index pages, and queries slow because they have to walk more series. We measure where the cardinality lives and either move those tags into fields or drop them at the Telegraf layer.

Should we move from InfluxDB 1.x to 2.x or wait for 3.x?

That depends on what you rely on. If your dashboards and tooling are built on InfluxQL and retention policies, staying on 1.x is a reasonable choice until support runs out. If you are already writing Flux or want better task management, 2.x is the right home. The jump to 3.x is worth planning but not rushing, because the storage engine and the ecosystem around it are still settling.

Can you take over an InfluxDB instance a previous team left us?

Yes. We inventory buckets, retention policies, tasks and tokens, document which services write and read where, rotate credentials, verify backups and replace any shared-user patterns with scoped tokens so the handover is clean.

How do we stop Telegraf from losing metrics when InfluxDB is busy?

Raise the buffer size with eyes open, set a reasonable flush interval, and add monitoring on the buffer high-water mark so you see the pressure before data is dropped. On the server side we usually also widen the write pool and smooth bursty inputs, because fighting the symptom alone just fills a bigger buffer slower.

Why do some of our Flux tasks fail silently?

Usually a query that occasionally crosses the task's timeout, or an upstream type that changes subtly and makes the Flux type system reject the script. Enabling task run logs and wiring their failures into the same alerting as the application is the usual first move; after that we rewrite the task to narrow its time window and avoid the implicit type guesses that bite on new data.

How much does InfluxDB help cost?

Quotes are free. Tell us the version, roughly how much data is in the instance and what has gone wrong, and we come back with a fixed price before any changes land.

Free estimate

InfluxDB giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.