Running MinIO?

MinIO cluster degraded or policies locked out? We ship it properly.

Self-hosted S3 looks cheap on paper until an erasure set drops below quorum at 2am, an IAM policy quietly forbids every uploader, or a rolling upgrade leaves half the pool on a version it will not talk to. Our engineers trace the ring, the lock daemon and the request log until each node agrees what it holds, then put monitoring around it so you hear about trouble before the dashboard turns red.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

minio-server.log

still broken

What it is

MinIO at a glance

MinIO is a high performance, S3-compatible object storage server first released in 2015 by the team at MinIO, Inc. It speaks the Amazon S3 REST API, so SDKs and tools built for AWS generally point at a MinIO endpoint with nothing more than a new hostname.

People run it on bare metal, Kubernetes and private cloud for backups, data lakes, model artifacts and anywhere the egress fees from a public cloud made the budget hurt. Features like server-side erasure coding, bucket replication, IAM compatible policies and KMS encryption line up with what people expect from S3, with the trade-off that you now own the drives, the ring and the upgrade calendar.

Type
S3-compatible object storage server
First released
2015
License
AGPLv3, with a commercial license for closed-source use
Maintained by
MinIO, Inc.
Typically used for
On-prem backups, data lake storage, AI model artifacts, private S3 drop-in

The honest version

MinIO pros and cons

No technology is all upside. This is where MinIO earns its place and where it tends to cause trouble.

Where MinIO is strong

  • Drop-in S3 APICode already written against AWS tends to work by swapping the endpoint. That makes migration from AWS, or a fallback for air-gapped sites, far cheaper than a bespoke storage layer.
  • No egress billPulling terabytes back out of the store costs network, not an invoice. For AI training sets and video pipelines this alone pays for the hardware in months.
  • Erasure coding without a separate licenseBuilt-in parity schemes protect against drive loss across a pool. You pick the shard count and the ring handles the rest.
  • Performance is genuinely strongOn NVMe, a decent pool will saturate 100Gb links. The read path is tight and the write path parallelises well across disks.
  • Works the same on your laptop and in productionA single binary runs standalone for development; the exact same binary scales out across nodes with a server pool argument.
  • Honest observabilityPrometheus metrics, audit logs and a trace endpoint make it possible to answer why a request slowed down without guessing at a black box.

Where MinIO bites

  • Upgrade windows are tightMixed-version pools refuse to talk after certain changes. Teams that upgrade one node at a time without reading the release notes end up with a cluster that will not form quorum.
  • AGPL has license implicationsShipping MinIO embedded in a commercial product is not free. The commercial license is fair, but it needs to be budgeted rather than assumed.
  • IAM is similar to S3, not identicalConditions, wildcards and resource ARNs differ in corners that only show up when a policy blocks a specific SDK call nobody tested.
  • Healing is heavyA failed drive triggers a background heal that competes with live traffic. Without rate limiting the latency on real requests climbs for hours.
  • Gateway and NAS modes were removedOlder deployments that used MinIO as a facade over NFS or another object store need a different plan now that those modes have been dropped from the server.
  • Hardware is now your problemFirmware, power, cooling, and spare drives all fall on your team. The savings versus AWS are real, but so is the pager.

Sound familiar?

Common MinIO problems we fix

The MinIO problems that bring people to us, and what you have once each one is fixed.

Erasure: read quorum lost for object

An erasure set dropped below quorum

Multiple disks in the same set went offline at once, often during a reboot that took longer than the health check tolerated, leaving reads blocked until enough drives reappear.

Pool back in quorum, heal watched

AccessDenied: s3:PutObject denied by bucket policy

A policy refuses a client that worked yesterday

A condition on source IP, a missing resource ARN pattern, or a stale service account key that the audit log pins down in under a minute.

Policy matches real traffic

SignatureDoesNotMatch

Signatures fail against a new deployment

A trailing slash in the endpoint, a region string that does not match, or a reverse proxy rewriting headers before the request reaches MinIO.

Clients sign and succeed

XMinioStorageFull

The pool is out of usable space

Soft quota reached, erasure overhead larger than the team expected, or lifecycle rules that never fired on the right prefix.

Headroom reclaimed, quotas tuned

Replication: target endpoint unreachable

Site replication has fallen behind

A VPN flap, a certificate that expired on the remote site, or a target version too old to accept the current replication protocol.

Replication caught up, lag watched

config version mismatch across nodes

Nodes refuse to rejoin after an upgrade

A partial rollout left different builds in the pool. The server refuses to form a set because the metadata layout changed between versions.

Cluster uniform, upgrade completed

Lock daemon: too many clients

Writers block each other under load

A workload that fans out hundreds of concurrent PUTs to the same prefix will exhaust the distributed lock daemon until the client count drops.

Lock pressure reduced, throughput restored

Drive heal required

A drive is scanned but never fully healed

Bitrot on a few objects, an aborted previous heal, or disks swapped without the correct metadata step leaving the pool reporting degraded indefinitely.

Heal finishes clean

What we do

MinIO help, from rescue to steady state

Pick up the phone for a single stuck heal, or hand the whole pool over for ongoing care.

Rescue
A cluster that will not form, a pool out of quorum, or policies blocking every uploader. We stabilize the ring first and write the postmortem second.
Deployment planning
Drive counts, erasure stripe sizing, network topology and KMS wiring worked out before the first node boots.
Policy and IAM review
Bucket policies, user policies and STS mappings audited against the SDK calls your applications actually make.
Replication and backup
Site replication across regions or buckets, with lag alerting that fires before compliance notices it.
Upgrades and migrations
Rolling upgrades rehearsed on a copy, and moves from AWS or Ceph onto MinIO without changing application code.
Managed Tech
Prometheus dashboards, heal watchers, drive-failure runbooks and a team on call when the pool starts complaining.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running MinIO alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

MinIO questions

Something else? Call+1 737 727 7327

Is MinIO really drop-in compatible with S3?

For the vast majority of SDK calls, yes. Where it differs is at the IAM edge: condition keys, some object lock semantics, and a few newer S3 features that lag behind AWS by a release or two. If your code sticks to GetObject, PutObject, ListObjects and standard multipart uploads, moving the endpoint is usually all the change you need.

Why did my erasure set lose quorum during a reboot?

MinIO waits a short grace period for drives to come back; if more drives in a single set stay offline past that window than the parity allows, reads for objects in that set block until enough drives return. The usual cause is rebooting too many nodes at once, or a storage controller that enumerates drives slowly after power-on. Staggering reboots and tuning the heartbeat fixes it.

Can MinIO replicate to AWS S3 or Backblaze?

Yes. Bucket replication speaks plain S3 at the target, so AWS, Wasabi and Backblaze B2 all work as destinations. The common gotchas are IAM on the far side, differences in storage class names, and clock skew when the replicator signs requests. Site replication between MinIO clusters is a separate feature with its own state.

Should we encrypt at rest with MinIO's built-in KES, or an external KMS?

KES is the right default for most self-hosted setups because it keeps the keys close to the storage. If you already run HashiCorp Vault or a cloud KMS and want a single source of truth for keys across systems, wiring MinIO into that is well supported. The decision usually turns on who holds the root key material.

How do we upgrade MinIO without taking the cluster down?

MinIO supports rolling upgrades inside a release train, but major version jumps occasionally change on-disk metadata and need a stop-the-world step. We rehearse the move on a staging pool with the same server-pool topology, confirm that mc admin update works across every node, and only then schedule the live upgrade with a rollback plan.

How much does MinIO help cost?

The estimate itself is free. Send us the server logs or a description of what the pool is doing and we come back with a fixed price before anyone touches production.

Free estimate

MinIO giving you trouble?

Tell us what the pool looks like and what the client is seeing, and a real engineer replies with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.