Running on Buildkite?

Buildkite agents offline or pipelines stuck? We sort it out.

Buildkite earns its keep when the agents stay healthy and the pipeline uploads behave. Teams usually arrive after a queue has been clogged for hours, a tag no longer matches any agent, or a dynamic pipeline is emitting YAML that will not parse. We walk the agent logs, the pipeline step by step, and the hooks around them until the real blocker is on the table.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

buildkite-agent.log

still failing

What it is

Buildkite at a glance

Buildkite is a hybrid continuous integration platform founded in Melbourne in 2013. The control plane and web UI are hosted by Buildkite, while the agents that actually run jobs live inside your own infrastructure, which is the reason many regulated and data-heavy teams reach for it.

Pipelines are defined in YAML, can be uploaded dynamically from a build step, and can be fanned out into thousands of parallel jobs using groups, matrices and wait steps. Agents are small Go binaries that register with a tag set and pick up work that matches. The hybrid model keeps source code, test data and deploy credentials on your side of the firewall.

Type
Hybrid CI with self-hosted agents
First released
2013
License
Proprietary SaaS control plane, MIT agent
Maintained by
Buildkite Pty Ltd
Typically used for
Monorepo CI, deploy pipelines, parallel test suites, regulated workloads kept inside a VPC

The honest version

Buildkite pros and cons

No technology is all upside. This is where Buildkite earns its place and where it tends to cause trouble.

Where Buildkite is strong

  • Agents live on your sideTest data, source code and deploy credentials never leave your VPC, which is a hard requirement for teams in finance, health and regulated SaaS.
  • Dynamic pipelines scale with the repoA pipeline step can print more YAML, letting one monorepo generate the right jobs for the paths that actually changed instead of running everything.
  • Parallelism is genuinely cheapYou pay per seat, not per minute, so fanning a suite out across hundreds of agents does not punish the budget the way other CI vendors do.
  • The agent is a small, boring Go binaryIt runs on EC2, bare metal, a Mac mini in the office or inside a container, which keeps the fleet choices in your hands.
  • Good plugin ecosystemDocker, Docker Compose, test splitting and artifact plugins cover most routine needs without writing shell by hand.
  • Clean UI for complex graphsThe build view handles wait steps, matrix fan-out and block steps without turning into a wall of noise, which matters on a pipeline with hundreds of jobs.

Where Buildkite bites

  • You own the agent fleetCapacity planning, autoscaling, image maintenance and OS patches are yours. The Elastic CI stack helps on AWS, but it is still infrastructure you keep alive.
  • Agent-tag mismatches are a classic foot-gunA step asking for a queue or tag that no live agent advertises will sit there forever rather than fail, so a typo costs an entire night of CI.
  • Pipeline upload errors appear at runtimeDynamic YAML generated by a script is not validated until the upload hits the control plane, so a bad interpolation only shows up once the build is already running.
  • Secret handling is on youThere is no managed secret store in the free tier; teams end up stitching together AWS Secrets Manager, Vault or S3 with the agent hooks, and getting the permissions right takes care.
  • Billing is per seat and climbs with the teamThe model is fair at a small team and starts to sting once contractors, offshore developers and bot accounts all need seats.
  • Observability is thinner than the hosted CIsFlaky test analytics, build insights and audit logs are extra products rather than part of the base plan, so budgeting for the whole picture matters.

Sound familiar?

Common Buildkite problems we fix

The Buildkite problems that bring people to us, and what you have once each one is fixed.

No agents matching tag queue=default

Builds are queued and no agent picks them up

A tag that was renamed last sprint, an autoscaler that never spun up, or a fleet that is sized for peak and currently asleep. Builds sit in the waiting state until somebody notices.

Queue drains, tags lined up with agents

failed to upload pipeline: invalid yaml

Dynamic pipeline uploads are breaking

A generator script emitting YAML with a bad interpolation, a duplicate key, or a step that references an environment variable that never existed on the uploading agent.

Pipeline upload clean on every branch

Agent lost connection, marking job as lost

Agents are vanishing mid-job

Spot instance reclaims, a machine running out of disk during docker pulls, or a security group change severing the long poll back to Buildkite.

Agents stable through the whole build

pre-checkout hook failed

Hooks break every new agent

A global hook that assumes a tool is on PATH, a credential file that only exists on one machine, or a hook that logs a secret into the plain build output.

Hooks portable and quiet

plugin docker#v5.8.0: manifest unknown

Plugin or image pulls fail intermittently

A tag that was overwritten upstream, a private registry that was never authenticated on the new fleet, or a plugin pinned to a version that has since been pulled.

Pulls pinned and authenticated

signed pipeline verification failed

Pipeline signing blocks a legitimate change

A key rotation that did not reach every agent, an upload path that was not included in the signature, or a mixed fleet where some agents have the new key and some do not.

Signing honoured across the fleet

Elastic CI stack scaled up: 0 to 12 agents

Autoscaling is either too slow or too eager

Cold starts of several minutes on every push, or a stack that never scales down and runs expensive instances overnight.

Scaling tuned to real patterns

no files found matching 'dist/**'

Artifact uploads silently produce nothing

A working directory that was reset between steps, a build that failed earlier without stopping the pipeline, or a glob that assumes paths the container never wrote.

Artifacts captured where they belong

What we do

Buildkite help, from agents to pipeline rescue

Ring us for one stuck queue, or hand us the whole CI estate.

Rescue
Queues stalled, agents offline or pipeline uploads failing on main. We unblock the fleet first and write up the cause second.
Agent fleet design
Elastic CI stacks, mixed queues for Linux and macOS, image bakes, spot and reserved mix balanced against your build hours.
Dynamic pipelines
Monorepo path filters, matrix fan-out, test splitting and block steps written so new services opt in without touching the core pipeline.
Secrets and signing
Secrets pulled from Vault or AWS at step time, signed pipelines rolled out safely, and agent hooks that do not leak into build logs.
Cost and concurrency
Seat audit, concurrency groups for shared resources, queue sizing, and autoscaler tuning so nights and weekends do not keep burning money.
Managed Tech
Fleet patching, upgrade windows, a dashboard pointed at queue times, and engineers on call when a Monday deploy stops moving.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Buildkite alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Buildkite questions

Something else? Call+1 737 727 7327

Why are my Buildkite agents offline?

The common reasons are a credentials expiry on the agent token, a networking rule that now blocks outbound to agent.buildkite.com, and spot instance reclaims that nobody noticed because the autoscaler never replaced the capacity. The dashboard shows last contact per agent and the agent log is usually explicit; the fix is almost always one of those three.

Why does my pipeline upload step keep failing?

Dynamic YAML is validated at upload time, not when you commit the generator script. A duplicate key, a Bash interpolation that produced an empty value, or a plugin reference whose version no longer exists will each kill the upload. Running the generator locally and piping the output to the pipeline parser catches most of them before they reach a build.

How do I secure secrets on a Buildkite agent?

Buildkite does not store secrets itself; it expects the agent to fetch them. Most teams pull from AWS Secrets Manager, Vault or S3 inside an environment hook, scope the IAM role to the queue, and set no-print on the variables so they are not echoed into the build log. Rotating the agent token on a schedule is also worth the small amount of effort.

Can you right-size the Elastic CI stack on AWS?

Yes, that is a frequent request. We look at build arrival patterns, warm pool sizes, instance families and the disk layout, then move the stack to a mix that holds a floor during business hours and scales to zero overnight. The before and after on the bill is usually visible inside a week.

How do I split a slow test suite across Buildkite agents?

Use the parallelism and concurrency keys on the step, combine it with a test splitter plugin, and feed the split by test timing rather than file count. Teams who skip the timing data end up with one shard running twice as long as the others, which wastes most of the saving.

How much does Buildkite help cost?

The estimate is free. Describe the fleet, the pipeline and what is blocking it, and we come back with a figure before any work is booked.

Free estimate

Buildkite giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.