Running HashiCorp Vault?

Vault sealed and nobody has the keys? We unstick it.

Vault is wonderful when it works and terrifying when it does not, usually at the worst possible moment. We take on the clusters that sealed themselves after a restart and nobody knows where the unseal keys are kept, the policies that are denying production access without a clear reason, and the auto-unseal setups that quietly stopped talking to KMS six months ago.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

vault-audit.log

still broken

What it is

HashiCorp Vault at a glance

HashiCorp Vault is a secrets management tool first released in 2015. It stores credentials, keys and other sensitive values, hands them out through authenticated sessions, and supports dynamic secrets that are generated on demand for databases, cloud providers and PKI. The core concepts are secret engines (kv, database, pki, aws, transit), auth methods (token, userpass, approle, OIDC, Kubernetes), policies (HCL that grants or denies paths) and leases.

Vault clusters typically run on integrated Raft storage these days, with auto-unseal against AWS KMS, Azure Key Vault or Google Cloud KMS so a restart does not require five humans with Shamir keys. The business model split in 2023 when HashiCorp moved to the BSL license, and OpenBao has emerged as the community fork. For teams on the licensed product, the operational shape has not changed.

Type
Secrets management, encryption as a service, PKI
First released
2015
License
BSL for Vault, MPL for OpenBao fork
Maintained by
HashiCorp (now IBM) and the OpenBao community
Typically used for
Application secrets, database credentials, PKI certificate issuance, cloud credential vending, SSH CA

The honest version

HashiCorp Vault pros and cons

No technology is all upside. This is where HashiCorp Vault earns its place and where it tends to cause trouble.

Where HashiCorp Vault is strong

  • Dynamic secrets change the whole modelShort-lived credentials issued on demand for Postgres, MySQL, AWS, Azure and GCP mean no long-lived keys sit anywhere, which is a real security improvement over static secrets in environment variables.
  • A single pane for many secret typesKV, database, PKI, transit, SSH, cloud credentials and transit encryption all run through one audit log and one access control system, which is tidier than wiring up five separate services.
  • Policies are fine-grained and readableHCL policies grant capabilities on paths, and the policy document is short enough to review in a pull request. Role-based access is expressible without a GUI full of checkboxes.
  • Encryption as a service without key material leavingThe transit engine encrypts and decrypts on request, so application code never sees a key. Key rotation, key versioning and derivation all work through the same API.
  • Good integration surfaceVault Agent, Kubernetes auth, AppRole, OIDC and the official Terraform provider cover most of how applications and operators interact with it, with sane patterns documented.

Where HashiCorp Vault bites

  • Sealed state is a sharp edgeVault starts sealed after any restart. Without auto-unseal set up correctly against a working KMS, you need a quorum of Shamir keyholders to bring it back, and those keys are often not where people think they are.
  • Operational burden is non-trivialA production Vault needs Raft nodes that are actually healthy, backups that are actually restored, audit logs on durable storage, and seals that are regularly tested. None of that is default.
  • BSL license changed the calculusThe 2023 license move means production use of HashiCorp Vault has terms that some legal departments want OpenBao instead, and the two codebases are only going to diverge over time.
  • Policies are easy to get wrongCapabilities, glob paths and parameter constraints combine in non-obvious ways. A permissive policy sits unnoticed, and a restrictive one breaks a release because nobody tested the production path.
  • Leases pile up if unmanagedDynamic secrets with aggressive TTLs can produce millions of leases that stress storage and the audit log, and cleanup of orphaned leases is a specific operational task rather than something Vault does on its own.
  • Upgrades across majors need rehearsalRaft snapshot formats, auto-unseal behavior and audit log schemas have changed between majors, and a rushed upgrade can leave a cluster unable to reach quorum.

Sound familiar?

Common HashiCorp Vault problems we fix

The HashiCorp Vault problems that bring people to us, and what you have once each one is fixed.

Vault is sealed

The cluster is sealed after a restart

A deploy, a kernel update or a cloud instance refresh took the nodes down and they came back sealed. Without auto-unseal or the Shamir shards, nothing talks to Vault and every application is now missing its secrets.

Cluster unsealed and durable

AWS KMS auto-unseal failed

Auto-unseal stopped working

An IAM policy that was tightened, a KMS key rotated, or a region change that left the Vault nodes without kms:Decrypt on the seal key. Vault is up but refuses to unseal because it cannot reach its seal.

Auto-unseal restored and tested

permission denied

A policy is denying production access

A policy rewrite that tightened paths more than intended, or an AppRole whose role_id is now mapped to the wrong set of policies. Applications fail with 403 and nobody is sure which path was lost.

Policies reviewed and documented

raft: no cluster leader

Raft has lost quorum

Two of three nodes lost at the same time, or a storage volume corrupted, and the remaining node cannot elect itself leader. Vault returns 500s and refuses writes until the cluster recovers.

Raft quorum restored, nodes healthy

token: no valid token found

Applications lost their token mid-run

A Vault Agent that stopped renewing because the renewable flag was never set, or a workload that cached a token past its TTL. Every request after the expiry returns a 403.

Token lifecycle handled by Agent

deprecation warning: kv v1

The secret engines are generations behind

KV v1 engines still in production, PKI engines without rotation, and transit configurations that predate current defaults. Not broken, but exposed when the next major removes support.

Engines migrated onto current versions

audit log write failed, disk full

Audit logging fell over

An audit device pointed at a filesystem that filled up, and Vault blocks requests because it will not serve without an audit trail. Everything stops until the log is writable again.

Audit logs durable and watched

pki: certificate not found: rotation did not run

PKI rotation stopped quietly

A root or intermediate CA whose rotation schedule was set up by somebody who left, and the job that renewed certificates stopped running. Services start failing on TLS expiry weeks later.

PKI lifecycle automated and alerted

What we do

Vault help, from rescue to upkeep

Call us for one sealed cluster, or hand us the whole Vault estate.

Rescue
A sealed cluster, a quorum loss, or a policy change that locked production out of its own secrets. We get Vault back online and then explain what caused the outage.
Auto-unseal and HA
Auto-unseal set up against AWS KMS, Azure Key Vault or Google Cloud KMS with the correct IAM, Raft cluster sized and tested for a node loss, and recovery keys stored where they actually belong.
Policy review
A pass through every policy and auth method against who is actually using them, with permissions tightened where they are too wide and relaxed where they are breaking releases.
Secret engine migrations
KV v1 to v2, legacy PKI rotated onto modern defaults, and dynamic database credentials introduced where static secrets are still sitting in environment variables.
OpenBao migration
For teams where the BSL license is an issue, a planned move from HashiCorp Vault onto OpenBao with policies, engines and auth methods carried across and tested.
Managed Tech
Seal monitoring, audit log watch, periodic unseal drills, upgrade rehearsals and a team to page when a cloud outage takes KMS offline.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running HashiCorp Vault alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

HashiCorp Vault questions

Something else? Call+1 737 727 7327

Our Vault cluster is sealed, what do we do?

If auto-unseal is configured and working, Vault unseals itself on restart; if it does not, something has changed between Vault and the seal (KMS key permissions, region, rotated key). If the cluster uses Shamir seal, you need a threshold of unseal keys from the keyholders. We work with you to either restore auto-unseal against the correct KMS key or reconstitute the Shamir quorum, document how, and set up alerting so the next seal is not a surprise.

How should we handle Vault unseal keys?

For anything beyond a toy install, you should be on auto-unseal against a cloud KMS, with the recovery keys stored in a documented, access-controlled secure location (a corporate password manager with M-of-N access, or a physical safe for the paranoid). Shamir keys in individual engineers' heads are how Vault clusters stay sealed permanently when somebody leaves.

Why is our Vault denying access after a policy change?

Policies use path globs and capability lists that are easy to narrow by accident. A policy that was supposed to allow read on secret/data/app/* but was written as secret/app/* will deny on a kv v2 engine. We diff the before and after, test the specific denied paths against the token's effective policies, and rewrite the policy so the application works and the audit log shows why.

Should we move from HashiCorp Vault to OpenBao?

The honest answer depends on your license posture and how far you have invested in HashiCorp-specific features. For teams that use the open-source feature set and have legal concerns about the BSL, OpenBao is a reasonable path; the API compatibility is good today. For teams on Vault Enterprise relying on features such as performance replication or namespaces, the move needs more planning, and in some cases it does not make sense.

Can you take over a Vault cluster we inherited?

Yes. We inventory every secret engine, auth method and policy, rotate the admin and recovery credentials, verify backups by actually restoring them in a staging cluster, and bring the cluster onto a supported version through the correct upgrade path. Then we document who holds what, so the next incident is not a scavenger hunt.

How much does HashiCorp Vault help cost?

Quotes are at no charge. Tell us the cluster size, the version, and the problem you are seeing, and we come back with a price before anyone touches a seal.

Free estimate

Vault giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.