Running Datadog?

Datadog invoice climbing or alerts all over the place? We sort it out.

Datadog is a brilliant platform and an expensive one, and most of the Datadog work we are called in on is about untangling the second half of that. Custom metrics multiplying, logs ingested at a rate that nobody reviews, alerts configured once and never tuned. We cut the noise, trim the bill, and leave you with dashboards somebody will actually read.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

datadog-config.log

still broken

What it is

Datadog at a glance

Datadog is a SaaS monitoring and security platform founded in 2010 by Olivier Pomel and Alexis Lê-Quôc, trading on the Nasdaq since 2019 under the ticker DDOG. It ingests metrics, traces, logs, synthetic tests, real-user monitoring, profiling, security findings and cloud cost data through a single agent and a long list of integrations.

The platform's scope is wide: infrastructure monitoring, APM, log management, network performance, serverless, database monitoring, continuous profiling, cloud security posture management, workload protection, Sensitive Data Scanner, and more. Pricing is per host, per million custom metrics, per gigabyte of logs, per span and per feature, which makes the invoice line-itemed to a degree few observability products match.

Type
Observability and security SaaS platform
First released
2012 public release
License
Proprietary SaaS, feature- and volume-based pricing
Maintained by
Datadog, Inc.
Typically used for
Infrastructure monitoring, APM, log management, synthetic tests, real-user monitoring, cloud security, cost management

The honest version

Datadog pros and cons

No technology is all upside. This is where Datadog earns its place and where it tends to cause trouble.

Where Datadog is strong

  • One agent, many signalsA single Datadog Agent collects metrics, traces, logs and process data from a host. Operationally that beats running a different collector per signal type.
  • A huge integration catalogueOver 700 integrations for AWS, Azure, Google Cloud, Kubernetes, popular databases, queues and SaaS tools. Most stacks light up with built-in dashboards within an hour.
  • APM that is actually usableService maps, flame graphs and trace search feel designed for debugging real incidents, not for a sales demo. Engineers like it, which is rare.
  • Powerful query languageTimeseries queries and log search support arithmetic, grouping, rollups and anomaly detection in a single expression, which is a step beyond what simpler tools offer.
  • Security and cost in the same paneCSPM, workload security, and Cloud Cost Management living next to the engineering signals removes the usual silos between security, ops and finance.
  • Good RUM and syntheticsReal user monitoring and synthetic tests cover the parts of the user experience that server-side APM cannot see on its own.

Where Datadog bites

  • The bill climbs quietlyCustom metrics, log indexing, APM hosts, RUM sessions and each additional feature are billed separately. A six-figure month is easier to reach than teams expect.
  • Custom metrics are sneakyEvery unique tag combination is a separate metric. A tag that includes a user ID or a request path can turn a single custom metric into millions overnight.
  • Log ingestion is not log indexingLogs arrive, and the choice of which to index for search is a per-pipeline decision that teams rarely tune until the invoice makes them.
  • The feature set is enormous and overlappingThere are often two or three features that could solve a problem, with different pricing and limits. Picking the right one needs familiarity, which newer teams lack.
  • Agent upkeep adds upAgent versions drift, integrations get deprecated, and a cluster-level agent on Kubernetes needs RBAC and configuration that somebody owns. None of this is hard; all of it needs attention.
  • Alert fatigue is the defaultThe default monitor templates are generous and often fire for things nobody actually wants paged on. Without discipline the on-call becomes background noise.

Sound familiar?

Common Datadog problems we fix

The Datadog problems that bring people to us, and what you have once each one is fixed.

custom metrics: 142k of 100k

Custom metrics ballooned

A tag like user_id or request_path was added to a metric, each unique value became its own custom metric, and the invoice now has a five-figure line for cardinality nobody intended to buy.

Cardinality back under control

monthly spend: $38,420 (prev $14,200)

The invoice tripled in a month

A new product was onboarded with every feature enabled, APM sample rate set to 100% on a chatty service, and log pipelines indexing debug-level logs that nobody reads.

Spend traced to resources, cut cleanly

Log indexing: 4.1 TB

Logs over the plan

A service writing JSON logs with large payloads, exceptions duplicated across microservices, or a new log pipeline that forwards everything without filtering. Indexed storage runs out.

Right logs retained, cost predictable

monitor snoozed 22 days

Alerts firing on everything and nothing

A monitor set up years ago, muted after one false positive, and never unmuted. Meanwhile new monitors fire on anomalies that are just weekend traffic.

Monitors owned, tuned, alive

APM: 94% at sample 1.0

APM is capturing more than it needs to

A service with an aggressive trace rate because somebody wanted the demo to look good, running at full sampling on millions of requests with no dynamic sampling in place.

Sample rate honest for cost and signal

Kubernetes agent: RBAC forbidden

Agents cannot see the cluster

A Datadog cluster agent deployed with incomplete RBAC, so pod metrics, events and tags never make it into Datadog. Dashboards look empty and nobody is sure why.

Cluster agent healthy and authorised

212-widget dashboard, 8.4s load

Dashboards nobody can use

A screen built over two years with every widget somebody wanted added, loading slowly and showing nothing legible on a laptop screen. Usable dashboards are short and deliberate.

Dashboards you can read at a glance

Cloud Cost: no credentials

Cloud Cost Management never wired up

Datadog's CCM module is paid for but was never connected to the AWS billing export, so the engineering team pays for a feature that could cut their AWS bill.

CCM live and informing changes

What we do

Datadog help, from invoice rescue to upkeep

Ring us for one noisy monitor, or hand us the Datadog estate and let us trim it properly.

Rescue
Invoice climbing, alerts firing on everything, dashboards unusable. We cut the obvious noise first and plan the rest.
Cost review
Line-by-line walk through custom metrics, APM hosts, log indexes and features, with a prioritised list of changes and the invoice impact of each one.
Cardinality hygiene
Tags audited for high-cardinality traps, aggregated metrics where raw ones were used, and a monitoring policy so the problem does not come back.
Log pipelines
Which logs are kept, which are indexed and which are archived to S3, with Logging without Limits and archive rehydration set up when it fits.
Alert design
Monitors with owners, severities, runbooks and rotations to PagerDuty, and the muted ones reviewed one by one to decide which should return and which should retire.
Managed Tech
A monthly review of spend, alert quality and agent health, with a human to call when the Datadog view disagrees with reality during an incident.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Datadog alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Datadog questions

Something else? Call+1 737 727 7327

Why has our Datadog bill tripled?

The common culprits are custom metrics ballooning because a high-cardinality tag was added, log indexing enabled on verbose debug logs, APM sample rate left at 100 percent on a chatty service, and features turned on during onboarding that nobody is actively using. We open the usage dashboard with you and trace each major line item to its source.

How do we fix custom metric cardinality?

Find the tags whose unique value counts are the problem, usually user_id, request_path or trace_id. Either aggregate the metric before it is sent, drop the tag at the agent or in the client library, or move the detail into a log where it belongs. The tools in Datadog's Metrics Summary make the offenders easy to spot once you know what to look for.

Should we use Datadog or Sentry?

Both, in most cases, with each one doing what it is good at. Sentry for application errors, source maps and session replay. Datadog for infrastructure, APM, logs and the business-wide dashboards. Overlap is small, and fighting to make one replace the other usually ends up costing more than running both.

Why are our Datadog alerts so noisy?

Default thresholds that were never tuned, monitors muted long ago and never unmuted, and lots of anomaly monitors on signals that are noisy by nature. We walk the monitor list with the on-call engineers and either tune, retire or route each one, which usually cuts pages by more than half.

Can you cut our log indexing costs?

Yes. We review each pipeline, move low-value logs to Logging without Limits so they are retained but not indexed, archive long-tail data to S3, and tune the parsers so high-volume log sources are deduplicated or dropped. Rehydration is available when the team needs to search historic data on a specific day.

How much does Datadog help cost?

The first review is free. Share the Datadog organisation and a recent invoice, and we come back with a written plan and a price before we touch any dashboard or monitor.

Free estimate

Datadog giving you grief?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.