Running Istio?

Istio sidecars misbehaving or mTLS blocking traffic? We unstick it.

Istio is powerful in the same way an aircraft cockpit is powerful. Every switch does something; most switches do several things; and nobody wrote down which combination the current deployment relies on. We go into real clusters where the sidecar is injecting into the wrong namespace, a PeerAuthentication is quietly blocking a database port, or an upgrade left half the mesh on one revision and half on another. We leave with a mesh someone else can actually operate.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

istio-proxy.log

still broken

What it is

Istio at a glance

Istio is a service mesh for Kubernetes and virtual machine workloads, originally jointly developed by Google, IBM and Lyft, and now governed as a CNCF graduated project. It attaches a sidecar proxy, by default Envoy, to each workload pod and routes all east west traffic through the mesh. That gives you mutual TLS between services, traffic shaping, retries, circuit breaking, authorization policy and deep telemetry without touching application code.

Modern Istio has an ambient mode that replaces the per pod sidecar with a shared ztunnel per node, cutting the overhead that made the mesh unpopular in smaller clusters. Alongside Linkerd and Consul Connect, Istio is one of the three mainstream mesh options in production use today.

Type
Service mesh with Envoy data plane
First released
2017 initial release
License
Apache 2.0, open source
Maintained by
CNCF graduated project with a wide contributor base
Typically used for
Service to service mTLS, traffic splitting, retries, authorization and telemetry in Kubernetes

The honest version

Istio pros and cons

No technology is all upside. This is where Istio earns its place and where it tends to cause trouble.

Where Istio is strong

  • Transparent mutual TLS between servicesService identities, certificate rotation and encrypted traffic between workloads come without application changes. Compliance teams stop having to ask about east west encryption.
  • Traffic management is first classVirtualService and DestinationRule give you weighted routing, canary releases, mirror traffic and retries declaratively. Blue and green deploys stop needing custom load balancer logic.
  • Observability for freeEvery hop gets metrics, traces and access logs. Kiali, Grafana and Jaeger slot in and give you a picture of the mesh most teams would not have built otherwise.
  • Authorization policy without library changesAuthorizationPolicy lets you express what services can call what endpoints, at the proxy, without an in app library. One policy language instead of per service rules.
  • Ambient mode cuts resource overheadThe new ambient architecture replaces per pod sidecars with a shared ztunnel per node, which lowers memory cost and avoids the pod startup race that sidecar injection sometimes loses.

Where Istio bites

  • Operational complexity is highIstio adds CRDs, webhooks, a control plane, proxies on every pod and a telemetry pipeline. Something going wrong in any of those layers can look like an application bug. Small teams do not always need this machinery.
  • Sidecar overhead is realMemory, CPU and startup time per pod add up at scale. Ambient mode mitigates it, but most production meshes are still on sidecars with the costs that come with it.
  • Upgrades are a planned exerciseCanary upgrades with revision labels are the supported path. Doing them wrong leaves half the pods on one revision and half on another, which is exactly as fun as it sounds.
  • Envoy configuration is a project in itselfWhen a user visible 503 lands, debugging means reading Envoy admin output, access logs and the generated configuration from pilot. That is not a skillset every team has.
  • mTLS strict mode breaks the quiet assumptionsSwitching to STRICT without permissive first will break anything talking HTTP from outside the mesh, legacy jobs, external dependencies, and anyone who forgot to label a port correctly.
  • Policy is powerful and dangerousAuthorizationPolicy has an implicit deny once any ALLOW is in scope. One wrong namespace selector can brick a whole production tier before anyone notices.

Sound familiar?

Common Istio problems we fix

The Istio problems that bring people to us, and what you have once each one is fixed.

sidecar injection webhook denied the request

Sidecars will not inject into new pods

A namespace missing the istio-injection label, a webhook failing because the control plane is on a different revision, or a pod template with an opt out annotation nobody remembered setting.

Sidecars injected consistently across the mesh

PeerAuthentication STRICT rejected request

mTLS is blocking legitimate traffic

A strict mode rollout that caught a client still speaking plain HTTP, or a non mesh workload that needs to reach a mesh service. Needs permissive mode plus observability before STRICT is turned back on.

mTLS enforced without breaking clients

503 NR: no route found

VirtualService routing is dropping requests

A hostname that does not match, a path prefix in the wrong order, or a VirtualService scoped to the wrong gateway. The 503 looks like a backend problem but is really a routing problem.

VirtualService routes resolved correctly

503 UC: upstream_reset_before_response_started

Envoy sees the upstream close the connection

A backend that panics, a liveness probe that kills the pod mid request, or an HTTP/2 flow control issue on a long response. Envoy logs point at the right place; we read them.

Upstreams stable, proxies happy

RBAC access denied

AuthorizationPolicy is quietly blocking a caller

A policy that selected more workloads than intended, or a missing rule for a caller that used to work through a different code path. We find the specific policy and widen or tighten it correctly.

Authorization matches intended access

istiod fails to push configuration

The control plane cannot reach proxies

Network policies blocking 15012 between proxies and istiod, a certificate rotation issue, or a pilot overloaded by too many push triggers. The whole mesh starts stuttering until it is fixed.

Control plane healthy, pushes landing

canary revision does not match control plane

A canary upgrade left the mesh in two states

Half the pods tag against revision 1-19, half against 1-20, and webhooks on one revision fight the other. Rolling the mesh forward or back has to be ordered deliberately.

Mesh on a single revision again

What we do

Istio help, from one bad route to a managed mesh

Reach us for one failing VirtualService, or let us run the mesh for you.

Rescue
mTLS blocking traffic, sidecar injection failing, or a mesh upgrade that left two revisions fighting. We stabilise and explain second.
mTLS rollout
Permissive mode first with telemetry to find the plain HTTP callers, policies written against real data, then STRICT turned on without losing requests in the switch.
Traffic management
VirtualService, DestinationRule and Gateway written for canary releases, blue green deploys, mirroring and retries that are safe rather than aggressive.
Ambient migration
Moving from sidecar mode to ambient where it fits, with capacity planning, telemetry parity and a rollback path at each step.
Observability
Kiali, Prometheus and distributed traces wired together so a 503 in the mesh tells you where it came from, not just that it happened.
Managed Tech
A standing arrangement covering Istio minor upgrades, Envoy CVE response, policy review and a mesh engineer available when a routing change at 11pm takes production down.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Istio alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Istio questions

Something else? Call+1 737 727 7327

Do we actually need Istio?

Often the honest answer is no. If you have a handful of services with modest security needs and one or two deploy patterns, Linkerd or no mesh at all is usually cheaper to operate. Istio earns its place when you have many services, many teams, strict mTLS requirements, or complex traffic shaping. We will tell you plainly when something simpler would serve.

Why is our Istio sidecar not being injected?

Usually the namespace is missing the istio-injection or istio.io/rev label, the pod has an opt out annotation, or the admission webhook is failing because the control plane is on a different revision. We check each layer rather than toggling labels and hoping.

How do we move from sidecar mode to ambient?

Carefully and in stages. We profile the current mesh, pick namespaces that benefit most from the move, migrate workloads gradually, and keep sidecars running on namespaces that need features ambient does not yet cover. Observability parity is the gating check at each stage.

Can you take over an Istio cluster from a previous team?

Yes. We inventory what revisions are deployed, what policies are active, which VirtualServices route traffic and who owns each workload. Then we bring it under version control, document the mesh and fix the drift before changing behaviour.

Why does our Istio upgrade keep going wrong?

The supported path is canary revisions with revision labels, and most failed upgrades skip that pattern. Doing an in place upgrade with mixed revisions left in the mesh is where it unravels. We run upgrades through a rehearsal cluster with the same workload shape before touching production.

How much does Istio help cost?

Quotes are at no charge. Share the Istio version, the symptoms and a kubectl describe output and we return a price before any changes land.

Free estimate

Istio giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.