Running OpenTelemetry?

OpenTelemetry Collector dropping spans? We fix it.

OpenTelemetry promises one instrumentation that goes anywhere, and the promise usually holds right up to the point where the Collector queue backs up, the Python auto-instrumentation disagrees with the Java service it is tracing, and the exporter to your backend starts returning 401 at 2am. We walk the pipeline from the SDK through the Collector to the exporter and remove the ambiguity.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

otel-collector.log

still broken

What it is

OpenTelemetry at a glance

OpenTelemetry is the vendor-neutral standard for generating, collecting and exporting telemetry data: traces, metrics, and logs. It formed in 2019 by merging OpenCensus and OpenTracing, two earlier projects, and now lives under the Cloud Native Computing Foundation as one of its most active communities.

The stack has three parts you typically touch. The SDKs, available for most mainstream languages, instrument applications and emit OTLP data. The Collector, a single binary written in Go, receives OTLP (and other formats), processes it with batchers, filters and samplers, and exports it to one or more backends. The protocols and semantic conventions are the glue that make it possible to switch backends without rewriting the instrumentation.

Type
Vendor-neutral observability framework for traces, metrics, and logs
First released
2019, merging OpenCensus and OpenTracing
License
Apache 2.0, open source
Maintained by
The OpenTelemetry community under the CNCF
Typically used for
Instrumenting services, aggregating telemetry, exporting to Datadog, New Relic, Jaeger, Honeycomb, Grafana Tempo and others

The honest version

OpenTelemetry pros and cons

No technology is all upside. This is where OpenTelemetry earns its place and where it tends to cause trouble.

Where OpenTelemetry is strong

  • Instrument once, export anywhereChanging observability vendor becomes a Collector exporter change rather than a rewrite of every service, which is a huge long-term cost saver.
  • Auto-instrumentation for common frameworksFlask, Express, Spring, Rails and .NET Core can be traced without touching the application code, which lowers the barrier for getting early value.
  • The Collector is a serious piece of infrastructureReceivers, processors and exporters compose into pipelines that handle batching, retries, filtering, enrichment and multi-backend fan-out.
  • Semantic conventions mean the data is interoperableAttribute names for HTTP, database, messaging and RPC are standardised, so dashboards and alerts travel between vendors.
  • Active community and real investmentNearly every observability vendor contributes, and the protocol and SDK work is moving fast rather than drifting.
  • Alternatives to proprietary agentsReplacing a Datadog or New Relic agent with the OTel Collector reduces per-host licensing and gives you a lever to negotiate with.

Where OpenTelemetry bites

  • The stack has a learning curveReceivers, processors, exporters, samplers, resource detection, context propagation and semantic conventions are all separate concepts. A team comes up to speed in weeks rather than hours.
  • SDK maturity is uneven across languagesJava and Go SDKs are mature. Some ecosystems, including PHP, Ruby and Erlang, are catching up but still have rough edges, particularly around auto-instrumentation.
  • Collector configuration is YAML all the way downPipelines, processors and exporters live in a config file that quietly grows past the point where one engineer understands it. Templating helps but is not standard.
  • Tail sampling costs memoryKeeping full traces in memory until the sampling decision is made puts real load on the Collector. Head sampling is cheaper but loses the interesting slow traces.
  • Exporter failures can silently drop dataA backend that rate-limits or rotates its token, and a Collector without a dead-letter queue, means telemetry disappears without the SDK noticing.
  • Context propagation breaks across boundariesA proxy that strips headers, a legacy service that uses B3 instead of W3C, or a message queue that does not forward context, each produce broken traces that look like a bug in the application.

Sound familiar?

Common OpenTelemetry problems we fix

The OpenTelemetry problems that bring people to us, and what you have once each one is fixed.

exporter: code = Unauthenticated

Export to the backend fails on auth

A rotated API key the Collector was never updated with, a token with the wrong scope, or an OTLP endpoint that changed path between major versions. Data piles up in the queue and then drops.

Export flowing to the backend again

memory_limiter: soft limit reached

Collector pods start refusing data

Batch sizes larger than the memory budget, a tail sampler holding too many traces in memory, or a backend slow enough to let the queue grow. Receivers start pushing back.

Collector running inside its budget

resource schema invalid

Backend rejects the payload

Resource attributes that do not match the semantic convention version the backend expects, often because an old SDK is emitting an older schema. The whole payload is rejected.

Schema aligned across the pipeline

traceparent header malformed

Traces fragment across service boundaries

An ingress that strips headers, a legacy proxy that uses B3, or a Kafka producer that does not propagate context. Each service shows its own spans with no parent, so no trace is complete.

Context propagated end to end

config references unknown processor

Collector refuses to start

A pipeline that names a processor the Collector binary was not built with (such as tail_sampling in a minimal distribution). The container restart loops with a confusing error.

Collector built with what it needs

429 Too Many Requests

Backend rate limits the exporter

A burst of spans from a deploy or a batch job exceeded the vendor rate limit, and the Collector's retry policy has not been tuned to back off accordingly.

Backoff that respects the backend

TLS handshake failed: unknown authority

Collector cannot reach a receiver securely

A self-signed certificate on a Jaeger or OTLP receiver that the Collector image does not trust, or an expired cert that nobody rotated.

TLS chain trusted and current

duplicate key 'service.name'

Resource attributes collide

An auto-instrumentation layer setting service.name alongside a resource processor that sets it again from the environment. One wins at random and the backend ends up with mixed identities.

Resource identity defined once

What we do

OpenTelemetry help, from one broken trace to the whole pipeline

Come to us with one lost trace, or hand us the whole telemetry pipeline.

Rescue
Spans vanishing, Collector OOMing, or exporter auth failing against a vendor. We get the pipeline back to healthy and explain what broke.
Collector design
Pipeline layout for the real traffic volume, sampling that captures the interesting traces, and dead-letter queues so a backend outage does not lose everything.
Instrumentation rollout
SDK integration for your actual services, auto-instrumentation where it is safe, and manual spans at the business boundaries that make the traces readable.
Vendor migrations
Moving from Datadog, New Relic or Honeycomb agents onto OTel without losing continuity, or running both in parallel while the switch is evaluated.
Context propagation
Ingress, service mesh, message queues and background jobs each wired to forward traceparent and baggage headers correctly, so traces stay whole.
Managed Tech
Ongoing Collector upgrades, backend swap tests, semantic convention updates applied, and a team on hand when a vendor rotates its auth at an awkward hour.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running OpenTelemetry alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

OpenTelemetry questions

Something else? Call+1 737 727 7327

Why is the OpenTelemetry Collector dropping data?

The three common causes are an exporter that cannot keep up with the receiver throughput, a memory_limiter processor pushing back when batches grow, and an authentication or TLS failure to the backend that the queue cannot drain past. We instrument the Collector's own metrics, find which stage is the bottleneck, and tune batch size, queue size, or exporter concurrency accordingly.

Should we use head sampling or tail sampling?

Head sampling is cheap, decided at trace start, and loses visibility into exactly the slow or error traces you want to see. Tail sampling waits until the trace ends, keeps the interesting ones, and costs real memory in the Collector. A common middle ground is head sampling at the SDK with a low ratio, plus always-sample rules for errors and high latency, which keeps cost down without dropping the traces you actually need.

Can OpenTelemetry replace a Datadog or New Relic agent?

Often yes, for traces and metrics. The caveat is that vendor-specific features, like Datadog's live tail, APM anomaly detection, or New Relic's distributed tracing UI quirks, may rely on data the OTel pipeline does not emit by default. We usually run both in parallel for a few weeks and compare before committing to the switch.

Why are my traces broken at service boundaries?

Something between services is not propagating the traceparent header. Common culprits are an ingress controller stripping custom headers, a legacy service still using the B3 format, a Kafka or SQS producer that does not forward context, or an HTTP client that was not instrumented. Each boundary needs to carry the context forward in a format the next service understands.

How do we keep the Collector config maintainable?

Three habits help. Keep the config in version control alongside other infrastructure, template it with Helm or similar so environment differences are isolated, and treat it like code with review, tests against a sample pipeline, and a canary deploy before rolling out to production Collectors.

How much does OpenTelemetry help cost?

A quote is on us. Share which languages are instrumented, what Collector topology you have, and which backend you are exporting to, and we respond with a price before any engineering starts.

Free estimate

OpenTelemetry giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.