Running Linkerd?

Linkerd proxy or certificate issues disrupting service? We sort it out.

Linkerd is the mesh people pick when they want the benefits without the cockpit. Most of what breaks for our clients is not configuration. It is a trust anchor nobody rotated, a proxy injector that never saw the namespace, or a viz extension that quietly lost its tap endpoint during a cluster upgrade. We go through the control plane, the proxies and the identity chain until the mesh is quietly doing its job again.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

linkerd-proxy.log

still broken

What it is

Linkerd at a glance

Linkerd is a service mesh for Kubernetes that pre dates Istio, originally built by Buoyant and now a CNCF graduated project. Linkerd 2 replaced the earlier Scala data plane with a purpose built Rust proxy called linkerd2-proxy, which is one of the reasons the mesh has a reputation for low resource overhead and predictable behaviour under load.

Compared to Istio, Linkerd trades configurability for operational simplicity. It covers the common case: automatic mutual TLS, golden metrics per service, retries and timeouts via ServiceProfile, and traffic splitting through the Gateway API. Features that Istio exposes through dozens of CRDs are either absent or kept intentionally opinionated. For many teams, that trade is the whole appeal.

Type
Service mesh with a Rust data plane
First released
2016 Linkerd 1, 2018 Linkerd 2
License
Apache 2.0, open source
Maintained by
CNCF graduated project, stewarded by Buoyant
Typically used for
mTLS between services, latency observability, retries and timeouts, lightweight mesh on Kubernetes

The honest version

Linkerd pros and cons

No technology is all upside. This is where Linkerd earns its place and where it tends to cause trouble.

Where Linkerd is strong

  • Low resource overheadThe Rust proxy is small and fast. Per pod memory and CPU costs are a fraction of what an Envoy sidecar adds, which matters once you multiply by hundreds of workloads.
  • Opinionated by designFeatures that would be configurable in Istio are chosen for you. The mesh is easier to operate because there are fewer wrong ways to configure it.
  • Automatic mutual TLSService identities, certificate issuance and rotation happen by default. You do not have to decide to turn mTLS on, only to rotate the trust anchor when it approaches expiry.
  • Good built in observabilityGolden metrics per service, per route tap and the viz extension give useful visibility without wiring up a separate stack for the basics.
  • Upgrades are smoother than IstioThe control plane and proxies have a saner compatibility story and clearer upgrade documentation. Multi revision rollouts are not the norm.

Where Linkerd bites

  • Less flexible than Istio when you need unusual routingWeighted routing, retries and timeouts are well covered. Anything more baroque, like header based routing across namespaces with complex authorization, is either harder or absent.
  • Trust anchor rotation is the operational trapThe root CA has a hard expiry. Teams that do not watch it will find their mesh stop issuing certs in production, with no easy workaround once it has lapsed.
  • Smaller ecosystem than IstioThird party tooling, dashboards and tutorials skew toward Istio. Not a problem for most workloads, but noticeable once you want niche integrations.
  • Policy model is narrowerThe authorization story has improved but is not as expressive as Istio's AuthorizationPolicy. Teams with complex security requirements may run out of room.
  • Fewer paid support optionsBuoyant offers enterprise support; the vendor market around Linkerd is smaller than Istio's, so leaning on a service provider is more of a direct Buoyant conversation.
  • Buoyant Cloud and Linkerd Enterprise are a separate considerationRecent versions of Linkerd have moved stable release channels behind an enterprise subscription. Teams that assumed everything stays free need to plan for that.

Sound familiar?

Common Linkerd problems we fix

The Linkerd problems that bring people to us, and what you have once each one is fixed.

trust anchor certificate will expire in 14 days

The root CA is about to expire

A trust anchor generated during install with a one year lifetime that nobody put on a calendar. Once expired, the mesh stops issuing new certs. Rotation needs to happen before, not after.

Trust anchor rotated and monitored

proxy-injector: config file does not contain valid key

The proxy injector webhook is failing

A secret that rotated and did not propagate, a namespace annotation missing, or an injector pod on a stale configuration after an upgrade. Workloads deploy without their proxy and the mesh gets quietly less meshy.

Proxies injected on every new pod

identity verification failed: certificate has expired

Workload certificates have expired

The identity service could not renew in time, or a pod has been running longer than the renewal window allowed. Traffic between the affected pods and the rest of the mesh starts failing.

Identity renewals healthy

linkerd viz tap: no live traffic observed

Viz is reporting zero traffic on live services

The viz extension lost its tap endpoint after an upgrade, the Prometheus pod has crashed, or RBAC on the viz namespace is blocking the queries. The mesh is working; the observability is misleading.

Metrics and tap back online

linkerd-cni: failed to install CNI plugin

Linkerd CNI is not initialising

A conflict with an existing CNI configuration, a CNI chaining order that puts Linkerd in the wrong place, or missing host path permissions. Pods start without the iptables redirect and bypass the proxy entirely.

CNI chain installed correctly

ServiceProfile retry budget exhausted

Retry budgets are firing on a hot endpoint

A ServiceProfile with an overly generous retry policy causing cascading retries on an already overloaded backend. Looks like the service is failing when the mesh is amplifying the load.

Retries conservative, backend stable

HA: replica count below minimum

Control plane high availability is degraded

Pods for identity, destination or proxy injector dropped below the HA minimum, usually because a node drained and the scheduler could not replace them in time. Control plane availability affects the whole mesh.

Control plane back to full HA

What we do

Linkerd help, from one broken proxy to a managed mesh

Reach us for one failing injector, or hand us the mesh for ongoing care.

Rescue
A trust anchor that expired, proxies that will not inject, or viz showing nothing on live traffic. We stabilise the mesh first and document the cause second.
Trust anchor rotation
Pre expiry rotation planned and rehearsed, with a monitoring hook so the next expiry never comes as a surprise.
Istio to Linkerd migration
Workload audit, feature parity gap analysis, and a staged migration that keeps both meshes running until each service has crossed over.
Observability and tap
Viz, Prometheus and Grafana wired so golden metrics, tap and topology actually reflect what the mesh is doing under load.
Policy and retries
ServiceProfiles, authorization policy and retry budgets written against real traffic so reliability features do not become reliability problems.
Managed Tech
A standing arrangement covering Linkerd upgrades, trust anchor monitoring, viz health and an on call engineer when the mesh starts failing certificate checks after hours.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Linkerd alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Linkerd questions

Something else? Call+1 737 727 7327

Should we pick Linkerd or Istio?

If you need lightweight mTLS, good observability and basic traffic management, Linkerd is usually the faster thing to live with. If you need complex traffic shaping, extensive authorization policy or heavy multicluster routing, Istio covers more surface area. We help teams run the comparison on their own workload rather than relying on blog posts.

Our trust anchor is about to expire. What now?

Rotate it before expiry, not after. The documented process involves issuing a new anchor, bundling both old and new during the transition, rolling the identity service and finally removing the old anchor. We have done this in production and can walk a team through it or run it ourselves.

Why is viz showing no traffic for a live service?

Usually the tap endpoint lost its service account token after an upgrade, the Prometheus pod crashed and nobody noticed, or the viz extension's RBAC was altered. The mesh itself is often fine; the observability stack is the problem.

Can we migrate from Istio to Linkerd?

Yes, and it is a common request after an Istio bill or operational incident. We audit the Istio features in use, check which have Linkerd equivalents, and migrate one namespace at a time with telemetry proving parity at each step.

Does Linkerd handle multicluster traffic?

Yes, through the multicluster extension. It is less flexible than Istio's model but easier to operate. We can set it up, test the failover behaviour and make sure certificate trust lines up across clusters.

How much does Linkerd help cost?

Estimates are free. Share the linkerd check output, the viz screenshot or a description of what is wrong and we come back with a figure before anything starts.

Free estimate

Linkerd giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.