Running Grafana?

Grafana dashboards blank or alerts misfiring? We sort it out.

Grafana reads from everything, which is its gift and its curse: a panel can go empty because the datasource timed out, the query hit a cardinality wall, the plugin was removed in the last minor release, or the folder permissions changed when someone else was cleaning up. We work through the real cause before we touch a single JSON model.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

grafana-server.log

still broken

What it is

Grafana at a glance

Grafana is an open-source visualization and alerting platform from Grafana Labs, first released by Torkel Odegaard in 2014 as a fork of Kibana. It reads metrics, logs and traces from dozens of backends: Prometheus, Loki, Tempo, InfluxDB, Elasticsearch, Postgres, CloudWatch, MySQL, and more through community plugins.

The core product is free and self-hosted, with Grafana Cloud selling a managed tier that bundles their LGTM stack (Loki, Grafana, Tempo, Mimir). Dashboards are stored as JSON, which makes them easy to version in Git and awkward to edit by hand. Unified Alerting replaced the old dashboard alerts in version 8, and the shift still trips up teams who inherited a mix.

Type
Metrics, logs and traces visualization
First released
2014
License
AGPLv3 (OSS), proprietary for Enterprise and Cloud tiers
Maintained by
Grafana Labs and a wide community
Typically used for
Infrastructure dashboards, SLO tracking, on-call alerting, business metrics, log exploration

The honest version

Grafana pros and cons

No technology is all upside. This is where Grafana earns its place and where it tends to cause trouble.

Where Grafana is strong

  • One pane across many backendsA single dashboard can mix Prometheus counters, Loki log counts, a CloudWatch gauge and a Postgres query, so the on-call engineer is not jumping between four consoles.
  • The OSS tier is genuinely freeYou can run production-grade Grafana on a small VM without paying for anything, and only reach for Cloud or Enterprise when you need the features.
  • Dashboards as codeJSON models and provisioning files put dashboards under version control, so a bad change can be reverted and new environments bootstrap without a human clicking.
  • Unified Alerting is powerful once it clicksRule groups, contact points, notification policies and silences cover most practical paging patterns, and the same rules work across datasources.
  • Huge plugin ecosystemPanels, datasources, apps and Scenes let you render almost anything, from flame graphs to geomaps to BigQuery dashboards, without writing the widget yourself.
  • Scales from laptop to global fleetThe same binary runs a homelab dashboard or a multi-tenant deployment with hundreds of users and SSO, just with different config.

Where Grafana bites

  • Query performance is the datasource's problem, not Grafana'sA slow dashboard almost always means a Prometheus series with too many labels or a SQL query without an index, and Grafana will cheerfully keep asking for it.
  • Alert rule debugging is awkwardThe Unified Alerting UI hides the actual rule evaluation, template expansion and routing decisions behind several clicks, and silent misroutes are the common failure.
  • Angular panels are being removedOlder dashboards still contain Angular panels and plugins that will not load after upgrading past Grafana 11. Migrating them is manual work that nobody wrote down.
  • Provisioning and the UI fight each otherEdit a provisioned dashboard in the UI and the next reload reverts your change. Edit it in Git and the UI shows a stale version. You pick one workflow per folder.
  • RBAC and teams only arrive in EnterpriseFolder-level permissions exist in OSS, but finer role-based controls, reporting, auditing and data source permissions require the paid tiers.
  • Upgrades can break plugins and panelsMinor versions drop signed plugin requirements, remove panels, or change the alerting datamodel. A ten-minute upgrade becomes a three-hour cleanup more often than the release notes imply.

Sound familiar?

Common Grafana problems we fix

The Grafana problems that bring people to us, and what you have once each one is fixed.

dial tcp 10.0.0.12:9090: i/o timeout

Panels are blank or showing N/A

A datasource that is unreachable from the Grafana pod, a security group that was tightened after an audit, or a Prometheus that is quietly out of memory. The dashboard looks broken; the problem is upstream.

Datasources reachable and queries returning

Contact point returned 429

Alerts fire but never page anyone

A notification policy that sends everything to one PagerDuty service until the vendor throttles it, or a Slack webhook that was rotated last quarter and nobody updated the contact point.

Alerts routed, delivered and acked

template: no value for key

Alert rules evaluate to 'no data' or 'error'

A legend format that references a label the series does not emit, a reduced expression with the wrong aggregation, or a datasource change that broke an inherited rule.

Rules green and honest

Plugin signature invalid

A plugin will not load after an upgrade

Grafana now requires signed plugins by default, and community plugins that were manually dropped into the folder refuse to load. Allowing unsigned plugins is possible, replacing them is better.

Plugins trusted or replaced

Dashboard provisioning: file not found

Provisioning changes are not showing up

A YAML path baked into the container image that no longer matches the mounted volume, or a syntax error in the provisioning file that fails silently at startup.

Provisioning applied and idempotent

sqlstore: database is locked

The Grafana instance is struggling under load

Default SQLite on a busy multi-user install, no caching on the datasource, or a single node trying to serve hundreds of concurrent viewers. The fix is Postgres or MySQL and some sizing.

Backend swapped, latency flat

OAuth failed: email claim missing

Users cannot log in via SSO

An identity provider that changed the claims it returns, a role mapping that does not match the new groups, or an auth proxy header that used to be trusted and no longer is.

Login restored, roles mapped

What we do

Grafana help, from one dashboard to the whole stack

Pick us up for a single broken alert, or let us run the whole observability tier.

Rescue
Dashboards empty, alerts silent, or the server refusing to boot after an upgrade. We stabilise the install first, then explain what the release notes did not.
Dashboards and datasources
Query performance looked at panel by panel, variables tidied up, folder permissions cleaned, and the JSON put into a repository where it can be reviewed.
Unified Alerting
Rule groups built against real SLOs, contact points tested, notification policies that route the right incident to the right team, and silences that do not hide real problems.
Upgrades and migrations
Moving from Grafana 8 or 9 to the current release, replacing Angular panels, migrating legacy alerts, and switching the backend from SQLite to Postgres when load demands it.
Grafana Cloud vs self-hosted
An honest look at whether the Cloud tier is worth the active series bill for your traffic, or whether self-hosting LGTM on your own cluster is cheaper and good enough.
Managed Tech
Weekly dashboard reviews, upgrade rehearsals, alert noise budgets and an on-call engineer who already knows your folders. Ongoing, not one-shot.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Grafana alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Grafana questions

Something else? Call+1 737 727 7327

Why are my Grafana panels showing 'No data'?

Three usual causes: the datasource cannot be reached from the Grafana server, the query returns nothing for the current time range, or the query is valid in a newer Prometheus and silently fails on the older one you are pointing at. Check the query inspector first, then test the datasource from inside the Grafana container, and only then suspect the panel itself.

Should we use Grafana Cloud or self-host?

Below a certain active series count, Grafana Cloud is cheaper than running your own Prometheus and Loki with storage, backups and on-call time included. Above that, self-hosting usually wins on cost but costs you a person to maintain it. We will work out the real break-even for your traffic before recommending either.

How do we migrate from legacy dashboard alerts to Unified Alerting?

There is a one-way migration tool, but the practical work is cleaning up what it produces: rule groups that make sense, contact points that are tested, and notification policies that match how the team is actually on-call. Doing it without the cleanup leaves you with a worse setup than you started with.

Our Grafana upgrade broke half the dashboards. What happened?

Most likely Angular panels being removed, plugins that were loaded unsigned, or alert rules that the migration did not convert cleanly. Rolling back is possible if the database was backed up before the upgrade. Otherwise the fix is panel by panel, which is tedious but not hard.

Can we put Grafana dashboards under version control?

Yes, and we recommend it. Dashboards exported as JSON, kept in Git, and reapplied via file provisioning or the API means a bad edit is a revert, and new environments stand up without a human clicking through the UI. The tradeoff is you stop using the UI to save changes, and that takes some adjustment.

How much does Grafana help cost?

The estimate is free. Send over the dashboards or a screenshot of what is broken, and we come back with a scope and a fixed price before anything in your instance is touched.

Free estimate

Grafana giving you trouble?

Dashboards, datasources, alerts, upgrades and SSO. If it is in your Grafana, we have probably pulled a similar one back together.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.