Observing with Honeycomb?

Honeycomb SLOs or derived columns wrong? We sort it out.

Honeycomb shines in the moments where other tools get vague, like a tail latency incident that moves between services, but it only pays back the investment if your traces carry the right attributes and your derived columns do not quietly return null. We work through the OpenTelemetry pipeline, the SLO definitions and the query shapes until the service map tells the truth.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

honeycomb-events.log

still failing

What it is

Honeycomb at a glance

Honeycomb is an observability platform built for high-cardinality, high-dimensionality data. It was founded in 2016 by Charity Majors and Christine Yen, and its central design idea is that debugging production is a query problem rather than a dashboard problem.

The platform takes events over the OpenTelemetry Protocol or its native Beeline SDKs, lets you slice them across arbitrary attributes, and surfaces anomalies with BubbleUp. SLOs are first-class and burn-rate alerting is well modelled. The pricing is event-volume based, which rewards teams who sample and tag thoughtfully and punishes teams who send everything and hope.

Type
Observability platform for traces, metrics and logs
First released
2016
License
Proprietary SaaS, event-volume pricing
Maintained by
Hound Technology, Inc. (Honeycomb)
Typically used for
Distributed tracing, high-cardinality debugging, SLOs, service level burn-rate alerting, BubbleUp for anomaly isolation

The honest version

Honeycomb pros and cons

No technology is all upside. This is where Honeycomb earns its place and where it tends to cause trouble.

Where Honeycomb is strong

  • High-cardinality fields are the pointGroup by user_id, feature flag, build sha or trace id without the metric-explosion penalty that traditional monitoring systems impose.
  • BubbleUp finds the signalComparing a slice of bad traces to a slice of good ones and highlighting the differential attributes often answers the 'why' in seconds rather than hours.
  • SLOs and burn-rate alerting are first-classDefining an SLI as a query, attaching an SLO to it, and alerting on burn rate is a few clicks rather than a side project.
  • OpenTelemetry-nativeYou can stick to vendor-neutral instrumentation and avoid the Beeline-only lock-in that older vendors still push.
  • Trace-first debuggingLooking at a slow request as a trace, then widening into the aggregate pattern, matches how engineers actually investigate incidents.
  • Reasonable free tier for small teamsThe free tier covers real workloads for small services, which makes adoption low risk.

Where Honeycomb bites

  • Volume pricing is a real constraintA service that over-instruments can burn through a plan quickly. Teams either tune sampling and tags, or pay for the surplus.
  • Instrumentation effort is yours to makeThe platform is only as good as the attributes your spans carry. A barely-instrumented service produces barely-useful queries.
  • Query result size limits catch newcomersGroup-by cardinality is excellent but has a limit, and queries that run against too wide a window return truncated results without a loud warning.
  • Beeline is sliding into legacy statusOpenTelemetry is the recommended path now, and Beeline-only codebases will need a migration.
  • Classic environments quietly fillAn older Honeycomb Classic dataset that was never migrated to the Environments model eventually hits its ceiling and stops accepting events.
  • Smaller integration ecosystemCompared to Datadog, there are fewer one-click integrations. For teams on exotic stacks, more of the integration work falls on them.

Sound familiar?

Common Honeycomb problems we fix

The Honeycomb problems that bring people to us, and what you have once each one is fixed.

field 'request.duration_ms' not found

Derived columns fail on missing fields

A derived column references an attribute that only some spans carry. Queries return null for the rest and dashboards show gaps that look like outages that are not real.

Derived columns defensive and working

SLO budget 12% remaining, no alert

SLO burn-rate alerts never fire

The SLO is defined but the burn-rate trigger was never wired up, or the escalation points at a channel nobody reads. The budget runs out before anyone looks.

SLO alerts firing to real humans

2.4M events dropped

Sampling is dropping events in bursts

A head-based sampler is set for steady state and cannot cope with a traffic spike, so a window of the incident is not in Honeycomb at all.

Sampling that survives bursts

schema change: user.plan type conflict

Schema conflicts break queries

A field that was a string in one service is being sent as a number by another, and Honeycomb refuses to merge the data cleanly. Queries that cross services return empty.

Schema aligned across services

span orphaned

Traces are missing spans and look incomplete

A worker service is not propagating the trace context, so background work shows up as separate root spans rather than children of the request that triggered it.

Context propagation complete

10,000 group-by limit

Group-by queries are being truncated

A query grouping by a very high-cardinality attribute hits the result limit, and the chart silently shows only the top groups.

Query shaped to the real question

Classic dataset full

A Classic environment stopped accepting events

A legacy Honeycomb Classic dataset was never migrated to the Environments model, hit its storage ceiling, and started rejecting new events.

Dataset migrated to Environments

What we do

Honeycomb help, from instrumentation to SLOs

Reach out for one noisy service, or let us own the observability practice end to end.

Rescue
Derived columns throwing errors, SLOs that never alert, or a sampling pipeline that is dropping events in the middle of an incident. We get the signal back first and clean up the pipeline after.
OpenTelemetry instrumentation
Picking the right span attributes, adding business tags such as tenant or feature flag, and getting trace context to propagate across every service and worker in your stack.
SLO definition and alerting
Modeling SLIs as queries that reflect the user experience, attaching sensible SLOs, and wiring the burn-rate alerts into the channels your team actually watches.
Sampling strategy
Choosing between head-based and tail-based sampling, keeping errors and slow requests always-on, and keeping the volume bill proportionate to the insight.
Migration from other observability tools
Moving from Datadog APM, New Relic or Classic Honeycomb onto OpenTelemetry-based Honeycomb Environments without losing historical visibility.
Managed Tech
Monthly reviews of the volume bill, SLO health and schema drift, plus an engineer who answers when an alert at 3am needs more than a dashboard glance.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Honeycomb alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Honeycomb questions

Something else? Call+1 737 727 7327

Why do our Honeycomb dashboards show gaps?

Common causes are a derived column that references a field only some events carry, a sampling policy that is dropping events during the window you care about, and a service that is sending with a different schema than the rest of your stack. We walk the query, confirm each contributing dataset, and fix either the column or the instrumentation at the source.

How do we keep our Honeycomb bill under control?

Volume pricing rewards sampling and thoughtful instrumentation. We implement tail-based sampling for the subset of traces that matter most, keep error and slow-request traces always on, and remove attributes that are neither useful for queries nor interesting for debugging.

Why isn't our SLO alerting us before we run out of budget?

Usually because the SLO is defined but the burn-rate trigger was never attached, or the trigger points at a channel nobody watches. We wire the alert to the real response path and prove it by injecting load that burns budget on purpose in a safe window.

Should we migrate from Beeline to OpenTelemetry?

Yes, eventually. Beeline is in maintenance and OpenTelemetry is where the Honeycomb product investment is going. We plan the migration service by service, run both in parallel during the overlap, and confirm span counts match before cutting over.

Can Honeycomb replace Datadog for us?

For tracing, high-cardinality debugging and SLOs, yes, and often better. For infrastructure metrics and the broader catalog of one-click integrations, it is a less complete replacement. Most teams end up with Honeycomb for applications and something lighter for the host-metric slice.

How much does Honeycomb help cost?

The estimate is on us. Share the dataset, the SLO or the schema issue, and we quote a fixed engagement before anyone logs into the environment.

Free estimate

Honeycomb giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.