Running HashiCorp Consul?

Consul gossip, ACLs or mesh routing causing outages? We sort it out.

Consul sits at the intersection of service discovery, configuration, secrets and mesh, which means it is always touched by something else that is breaking. A gossip partition caused by a flaky network, an ACL token that was never persisted across restarts, or a Connect intention nobody remembered writing. We work from the agent logs and the server state outward until the catalog matches reality and the mesh stops dropping traffic.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

consul-agent.log

still broken

What it is

HashiCorp Consul at a glance

Consul is HashiCorp's service networking platform, covering service discovery through a shared catalog, distributed key value storage, health checks, service mesh through Consul Connect, and multi datacenter federation. It runs as an agent on every node, with a small cluster of servers running Raft for the authoritative state.

Consul predates most of the Kubernetes native tools and remains popular in mixed estates where workloads run on virtual machines, bare metal and containers at the same time. Connect brings mTLS, authorization intentions and a sidecar mesh to the same catalog, with Envoy as the default data plane. Vault often sits alongside, Terraform usually feeds configuration, and Nomad often handles the workload scheduling if Kubernetes is not in the picture.

Type
Service discovery, key value store, service mesh
First released
2014 initial release
License
Business Source License since Consul 1.17, previously Mozilla Public License 2.0
Maintained by
HashiCorp, now part of IBM
Typically used for
Service discovery, DNS based catalog, mTLS mesh, config and secrets in mixed estates

The honest version

HashiCorp Consul pros and cons

No technology is all upside. This is where HashiCorp Consul earns its place and where it tends to cause trouble.

Where HashiCorp Consul is strong

  • One tool across virtual machines, bare metal and containersNot Kubernetes specific. The same agent runs on an old RHEL host and a modern container platform with the same model.
  • DNS interface works everywhereService lookups come out as SRV and A records. Legacy applications that cannot be rewritten for a modern discovery API still benefit from the catalog.
  • Catalog, KV, health and mesh share one agentFewer moving pieces than running separate service discovery, config, and mesh systems. Operational surface is smaller once the agent model is understood.
  • Multi datacenter federation is a first class featureCross datacenter service lookups, WAN gossip and prepared queries are built in. Teams running globally do not have to bolt federation together.
  • Integrates cleanly with Terraform, Vault and NomadThe HashiCorp stack is coherent. Configuration flows from Terraform into Consul, secrets come from Vault, and Nomad workloads register themselves automatically.

Where HashiCorp Consul bites

  • The licensing picture has changedConsul moved from MPL 2.0 to the Business Source License in version 1.17. Some teams have paused upgrades until they understand the implications, and the OpenTofu style fork conversation sits in the background.
  • ACLs are easy to get wrongThe system is powerful and unforgiving. A token missing a policy, a bootstrap token lost after a restart, or a legacy ACL mode left running will break the cluster in ways that look like network problems.
  • Gossip is sensitive to network conditionsSerf gossip relies on timely UDP. Noisy networks, aggressive firewalls and cloud virtual networks that drop packets can cause spurious member failures and route storms.
  • Connect adds another mesh to your lifeIf you also run Istio or Linkerd, Consul Connect is a third mesh in the environment. Picking one is cheaper than running both, and rarely done from day one.
  • Snapshot and backup discipline is criticalThe server state is small but precious. Clusters that do not snapshot regularly cannot recover cleanly from a majority loss, which is survivable only if you planned for it.
  • Raft and autopilot need attention on upgradesServer upgrades, especially across major versions, need the Raft cluster to maintain quorum at each step. Done without rehearsal, it is where most of the production incidents originate.

Sound familiar?

Common HashiCorp Consul problems we fix

The HashiCorp Consul problems that bring people to us, and what you have once each one is fixed.

agent: failed to sync remote state: ACL not found

Agents cannot authenticate to the catalog

An ACL token that was never persisted into the agent configuration, or a token whose policies changed under it. Agents go quiet and services disappear from the catalog.

Agents authenticated against the catalog

serf: 4 of 7 members unreachable

The gossip pool has fractured

UDP loss between nodes, a security group that dropped traffic on the serf port, or a noisy network causing cascading EventMemberFailed storms. Members flap and the catalog stops converging.

Gossip stable, membership accurate

raft: Failed to make RequestVote RPC

The Raft cluster cannot elect a leader

A dead server that autopilot never cleaned up, network reachability dropping below quorum, or a data directory on one server that fell out of sync. Writes to the catalog stop until a leader exists.

Raft quorum healthy, leader elected

connect: intention check failed

Service mesh intentions are blocking traffic

An explicit deny rule, or the default deny posture catching a caller that was never granted access. The mesh is doing exactly what the intentions say, even when that was not what was meant.

Intentions match intended access

health check failed

Services are flapping in and out of the catalog

A TCP or HTTP check with aggressive timing, a service that genuinely takes a few seconds to answer, or a probe that hits a non ready instance. Failures cascade into DNS and mesh behaviour.

Checks honest about readiness

upgrade: Raft protocol mismatch

A rolling server upgrade broke quorum

Raft protocol version changed between major releases, and the upgrade step order did not account for it. Half the servers are speaking one protocol and half another.

Cluster on a single Raft protocol

mesh gateway: failed to resolve SNI target

Cross datacenter mesh traffic is dropping

Mesh gateway SNI routing that does not match the service name, or a datacenter prefix missing in federation. Looks like a service outage; is really a routing config issue.

Cross datacenter routing resolved

What we do

Consul help, from gossip storms to steady state

Reach us for one dead server, or let us run the Consul estate on retainer.

Rescue
A cluster that lost quorum, ACL tokens nobody can recover, or a Connect intention blocking real traffic. We stabilise and explain what changed.
ACL and policy cleanup
Legacy ACL mode retired, tokens brought under a template or Vault issued model, and policies written against the services they protect rather than against history.
Server upgrades
Rehearsed Raft protocol upgrades, autopilot configured, snapshot cadence proven, and a rollback plan written down before the first live server is touched.
Connect mesh rollout
Sidecar deployment, intentions designed from service behaviour, and mesh gateways for federation that route correctly before STRICT is ever enforced.
Multi datacenter federation
WAN gossip tuning, mesh gateways between datacenters, prepared queries and a monitoring layer that notices federation drift before users do.
Managed Tech
A standing agreement covering Consul version upgrades, snapshot and backup discipline, ACL hygiene and an engineer on call when gossip gets noisy at 3am.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running HashiCorp Consul alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

HashiCorp Consul questions

Something else? Call+1 737 727 7327

Our Consul cluster lost quorum. What now?

The honest answer depends on whether the data directories on the surviving servers are intact. If they are, autopilot cleanup and a careful server rejoin usually bring quorum back. If they are not, you are restoring from the last known good snapshot, which is why snapshot discipline matters before anything goes wrong. We walk teams through both paths.

Should we move from Consul Connect to Istio or Linkerd?

If the only reason you have Consul is service discovery, and you also run Kubernetes with Istio or Linkerd, consolidating onto the Kubernetes mesh often simplifies the estate. If your workloads are mixed between virtual machines, bare metal and containers, Consul's catalog and mesh model is still a strong fit. We help with the honest comparison.

How do we handle the Business Source License change?

If you are on Consul 1.16 or earlier, you are on the old license and the question is whether and when to upgrade. For many teams the BSL terms are fine for production use. For some, they are not. We help read the terms against your specific case rather than offering legal advice.

Why do our ACL tokens stop working after a restart?

Usually the token is set on the agent via the API rather than persisted to configuration, so it survives until the process restarts and no further. Tokens written into the agent's ACL configuration section or injected from Vault survive restarts cleanly.

Can you set up multi datacenter Consul federation?

Yes. WAN gossip, mesh gateways and prepared queries are the three main pieces. Federation across cloud providers or on premise to cloud takes network work in addition to Consul configuration, and we usually run it with a staged rollout and a monitoring plan.

How much does Consul help cost?

Quotes do not cost anything. Share the consul members output, the version and a note on the symptom and we come back with a price before any work is scheduled.

Free estimate

Consul giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.