The cluster went red
A primary shard with no healthy copy, often after a disk watermark on a hot node, a stale replica after a network blip, or a corrupted index that cannot pass the recovery check.
Cluster green, shards assigned
Running OpenSearch?
A domain that lived at green for a year has slipped to yellow, Index State Management policies are stuck in the warm transition because of a shard allocation rule nobody remembers, and heap is sitting at ninety percent while bulk rejections climb at the coordinator layer. We open the allocation explain output, read the ISM history, and walk the cluster back to green without a weekend cutover.
What it is
OpenSearch is an open-source distributed search and analytics engine, forked by AWS from Elasticsearch 7.10 in 2021 after the license change to SSPL. It kept the Lucene-based inverted index, the near-real-time ingest model, the JSON document surface and most of the REST API, and it has since grown its own feature set including Index State Management, a native security plugin based on Open Distro, and an anomaly detector built on random cut forests.
The project lives under the OpenSearch Foundation, with a wide group of maintainers beyond AWS, and is the engine behind Amazon OpenSearch Service. Trouble in production clusters around the same handful of patterns as classic Elasticsearch, including JVM heap pressure, shard counts that have crept out of hand, mappings that explode on dynamic fields, and ISM policies whose transitions wait on an allocation rule that no longer makes sense. On the managed AWS service there are extra moving parts: blue-green deploys, master node failovers, and manual snapshots to S3 whose IAM role was never retested.
The honest version
No technology is all upside. This is where OpenSearch earns its place and where it tends to cause trouble.
Sound familiar?
The OpenSearch problems that bring people to us, and what you have once each one is fixed.
A primary shard with no healthy copy, often after a disk watermark on a hot node, a stale replica after a network blip, or a corrupted index that cannot pass the recovery check.
Cluster green, shards assigned
A rollover condition that no longer fires because the ingest rate dropped, or a warm tier whose allocation rule is unsatisfiable, so the policy loops without ever completing.
Policy conditions matched to real ingest
A dashboard asking for too many buckets at once, an aggregation on a text field, or a bulk request whose payload blew past the breaker. The breaker is correct; the shape of the work is wrong.
Queries and bulk sizes within heap limits
A disk watermark that triggered a cluster-level block, or a limit on the number of shards per node that was reached after unmanaged index creation.
Headroom back, blocks released
A node that briefly lost quorum and came back behind, or a replica that drifted during an index that rapidly rolled over. The cluster isolates the stale copy, which can leave fewer replicas than planned.
Stale copies recovered, replication healthy
On the managed service, a blue-green deploy or an instance replacement that took longer than the timeout. For a few minutes clients saw errors that look unrelated to the actual cause.
Client retries aligned to real failover times
A large manual snapshot that hits S3 request limits on a cold bucket, or a repository whose prefix is shared with other high-traffic data. Backups partially succeed and quietly fail retries.
Snapshot schedule and prefix tuned
A geoip or enrich processor calling a service that is no longer reachable from the data nodes after a VPC change, so every document hits an error and goes to the dead-letter path or is dropped.
Pipelines self-contained or routed correctly
What we do
Ring us for one red cluster, or let us own the search and logging stack.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
Yellow means at least one replica shard is unassigned, which is often tolerable on a small cluster and a warning on anything you care about. Red indicates a missing primary and therefore an availability problem until the shard is placed. The allocation explain API will tell you which shard and why, and the root cause is usually a disk watermark, a departed node, or an allocation rule that no longer resolves after a resize or an index template change.
That is a real decision with real tradeoffs. OpenSearch suits teams that want Apache 2.0 licensing, deep AWS integration and security without a paid tier. Elastic suits teams that want the newer vector and ML features, the official support channel, or the latest ILM and ES|QL work. We model the actual cost and feature gap against your workload rather than picking a side on principle.
Start by measuring what is consuming it. The usual suspects are fielddata on analyzed text, aggregations producing enormous bucket counts, and bulk payloads that blow past the request limit. A heap sized above roughly 31 gigabytes stops paying for itself because compressed object pointers switch off, so throwing memory at the problem rarely helps. Most heap pressure clears up when a handful of problem queries get rewritten, keyword fields replace text fields for aggregations, and the shard count is brought down to a figure the master can actually schedule.
Yes. We inventory the domain, the ISM policies, the index templates, the clients that write and read, and the IAM roles that back the snapshots. We rotate access, bring the configuration into Terraform or CloudFormation, and verify snapshots restore into a staging domain before changing anything.
Not in a single step. We check deprecation logs on the current version, rehearse the upgrade on a snapshot restored into a sandbox, confirm every ingest pipeline and client library works on the target version, deprecate removed APIs in index templates, and then schedule the real upgrade in stages with a rollback plan ready.
The estimate is always free. Describe the domain, what it is indexing and what has gone wrong, and we send back a price before any changes land.
Free estimate
Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: