Working with Terraform?

Terraform state locked or plan full of drift? We clear the backlog.

Terraform is a sharp tool and most of the pain we are called in to help with is self-inflicted: a state file on someone's laptop, a module that grew into a monolith, a provider upgrade that re-plans half the estate. We untangle the state, agree what is real, and leave the team with plans that mean something again.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

terraform-plan.log

still failing

What it is

Terraform at a glance

Terraform is an open-source infrastructure-as-code tool created by HashiCorp and first released in 2014. You describe cloud resources in HCL, Terraform asks the provider what exists today, works out the difference, and applies the plan. The state file records what Terraform believes it owns and is the single most important artifact in any Terraform project.

HashiCorp changed the license from MPL 2.0 to the Business Source License in August 2023, which prompted the community fork OpenTofu that launched in January 2024. Many teams now choose between Terraform, OpenTofu, and alternatives like Pulumi, with providers for AWS, Azure, Google Cloud, Kubernetes, GitHub, Cloudflare and hundreds more. Remote state in S3 or Terraform Cloud is the default for any team working in parallel.

Type
Infrastructure as code, declarative provisioning
First released
2014
License
Business Source License since v1.6 (was MPL 2.0)
Maintained by
HashiCorp, Inc. and a wide provider ecosystem
Typically used for
Cloud provisioning on AWS, Azure, Google Cloud, Kubernetes manifests, SaaS configuration like GitHub and Cloudflare, platform engineering

The honest version

Terraform pros and cons

No technology is all upside. This is where Terraform earns its place and where it tends to cause trouble.

Where Terraform is strong

  • A single description for many cloudsThe same HCL language, with different providers, describes AWS, Azure, Google Cloud, Kubernetes and dozens of SaaS products. Teams only have to learn one grammar.
  • Plans you can readterraform plan prints what will change before anything changes. Reviewers can see the blast radius of a pull request in the diff, not after a deploy.
  • A huge provider catalogueOver 3,000 providers in the Terraform Registry cover almost every public cloud and SaaS. If something has an API, there is probably a provider for it.
  • State locking stops two engineers from fightingRemote state in S3 with DynamoDB, in Terraform Cloud, or in OpenTofu's remote state, serialises applies and prevents the classic concurrent-write disaster.
  • Modules compose cleanlyReusable modules let a team build their own paved road: a VPC module, an EKS module, an RDS module, each with sensible defaults and tested inputs.
  • Drift detection is built interraform plan surfaces resources that have changed outside Terraform, so a console edit is not invisible for long if plans are run regularly.

Where Terraform bites

  • State is a liability as well as an assetLose the state file, corrupt it, or let it drift and you are negotiating with the cloud to rebuild it. Many of the worst incidents in Terraform projects trace back to state.
  • The licence change split the communityThe move to the BSL caused a fork to OpenTofu. Projects now have to pick, and tooling vendors have to support both, which adds a decision nobody wanted to make.
  • Provider upgrades can be painfulA major version of the AWS provider often re-plans schema in ways that touch dozens of resources. Even careful upgrades need rehearsal on a copy of the state.
  • Cycles and implicit dependencies catch beginnersA reference between two modules in the wrong direction produces a cycle error with a stack of resource names. Understanding depends_on and how to break cycles takes practice.
  • No native secrets managementTerraform reads secrets from somewhere and leaves them in the state in plain text. Storing state on S3 without encryption, or sharing state through Git, is a real risk teams still create.
  • Debugging failed applies is not funPartial applies leave infrastructure half-built, taint markers need manual clean up, and the error messages from providers range from clear to cryptic depending on the resource.

Sound familiar?

Common Terraform problems we fix

The Terraform problems that bring people to us, and what you have once each one is fixed.

Error acquiring the state lock

The state is locked and nobody knows by whom

A previous apply crashed, a CI runner was killed mid-run, or two engineers started plans at the same time. The lock is still held and every subsequent apply refuses to start.

Lock cleared, lineage verified

Objects have changed outside of Terraform

Drift between code and reality

Someone made changes in the AWS console during an incident, an auto-scaler changed a tag, or a different team updated a resource through a different pipeline. The plan now wants to undo the fix.

State and code back in agreement

Cycle: module.a <- module.b <- module.a

The plan errors with a cycle

Two modules reference each other, or a resource references an output that depends on itself through a chain. The dependency graph cannot be built.

Modules refactored, cycle gone

Error: Saved plan is stale

A saved plan no longer applies

The state has moved on since the plan was captured, so the recorded plan cannot be used. Usually this means two pipelines are running against the same state without coordination.

Plans pinned to current state

BucketAlreadyOwnedByYou

A resource exists but is not in state

Something created by hand before Terraform owned it, or a state file that was wiped and now lies about what Terraform thinks it manages. Importing is the usual answer.

Resource imported, state consistent

Provider produced inconsistent result

Provider bug after an upgrade

A provider minor version that changed how an attribute is computed, or an EC2 resource whose returned value does not match what the plan expected. Worth checking the provider changelog before swearing at it.

Provider pinned, workaround in place

47 resources changed, 12 destroyed

A plan wants to destroy production resources

A variable renamed and the old resources no longer tracked, a count that changed from a list to a for_each, or a module source bump that re-evaluates addresses.

Plan honest, destructive changes vetoed

state on a laptop

The state file lives on one engineer's machine

A terraform.tfstate committed to Git long ago, or sitting in a developer's home directory. Nobody else can apply, and losing the laptop loses the infrastructure.

Remote state with locking, backed up

What we do

Terraform help, from rescue to steady state

Call us for one stuck apply, or let us refactor the whole stack.

Rescue
Locked state, broken plans, half-applied changes. We stabilise the state file first and work backwards from there.
State hygiene
Migrating off a laptop, into S3 plus DynamoDB or Terraform Cloud, with encryption, backups and locking in place from day one.
Drift and import
Resources created outside Terraform imported into state, drift reconciled, and the console locked down so it does not happen again.
Module refactor
Monolithic root configurations split into modules with clear inputs and outputs, tested with terraform plan on a non-production copy.
Provider upgrades
AWS, Azure or Google provider major upgrades planned on a copy of the state, incompatible changes resolved before the real run.
Managed Tech
Steady plans on a schedule, provider upgrades tracked, state backups verified, and a human to call when an apply misbehaves after hours.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Terraform alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Terraform questions

Something else? Call+1 737 727 7327

Our Terraform state is locked, how do we clear it?

First confirm no one is actually running a plan, then look at the lock info to see who claimed it and when. If the lock is genuinely stale, terraform force-unlock with the lock ID clears it, and we always take a snapshot of the state before doing so. The real fix is making sure CI runners cannot be killed mid-apply, since that is where most stale locks come from.

Why does our plan want to destroy half of production?

Almost always a variable, count or for_each change that moved every resource to a new address. Terraform cannot tell the resources are the same thing, so it reads the old ones as orphaned and wants to destroy them. The repair is to move resources in state, not to apply the plan. We write out the exact terraform state mv commands before anything is applied.

Should we move to OpenTofu?

OpenTofu is a drop-in fork for most projects today and the community is growing. Teams that want to stay on the Business Source License for a managed product like Terraform Cloud keep Terraform; teams uneasy about the licence move to OpenTofu. We will give you a direct answer based on your providers, your CI setup and your risk tolerance.

How do we handle drift between Terraform and the console?

By running plans on a schedule rather than only before changes, so drift is visible within a day instead of a quarter. Resources that must change in the console during incidents are documented, and we either import the change back into state or revert it on purpose. Lifecycle ignore_changes is a tool, not a hiding place.

How do we upgrade the AWS provider from v3 or v4 to v5?

We pin the current version, read the upgrade guide carefully, apply the upgrade on a copy of the state against a non-production account, and resolve every re-planned resource with either a state move or a code change. Only when the plan is empty on the copy do we repeat on production, in a change window with a backup of state in hand.

How much does Terraform help cost?

The first conversation and the estimate are both free. Send a redacted plan, the state backend you are using, and what is going wrong, and we return a scope and a number before any apply runs.

Free estimate

Terraform giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.