Running OpenSearch?

OpenSearch cluster yellow, red, or crawling? We unstick it.

A domain that lived at green for a year has slipped to yellow, Index State Management policies are stuck in the warm transition because of a shard allocation rule nobody remembers, and heap is sitting at ninety percent while bulk rejections climb at the coordinator layer. We open the allocation explain output, read the ISM history, and walk the cluster back to green without a weekend cutover.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

opensearch.log

still broken

What it is

OpenSearch at a glance

OpenSearch is an open-source distributed search and analytics engine, forked by AWS from Elasticsearch 7.10 in 2021 after the license change to SSPL. It kept the Lucene-based inverted index, the near-real-time ingest model, the JSON document surface and most of the REST API, and it has since grown its own feature set including Index State Management, a native security plugin based on Open Distro, and an anomaly detector built on random cut forests.

The project lives under the OpenSearch Foundation, with a wide group of maintainers beyond AWS, and is the engine behind Amazon OpenSearch Service. Trouble in production clusters around the same handful of patterns as classic Elasticsearch, including JVM heap pressure, shard counts that have crept out of hand, mappings that explode on dynamic fields, and ISM policies whose transitions wait on an allocation rule that no longer makes sense. On the managed AWS service there are extra moving parts: blue-green deploys, master node failovers, and manual snapshots to S3 whose IAM role was never retested.

Type
Distributed search and analytics engine, Elasticsearch fork
First released
2021 (OpenSearch 1.0)
License
Apache 2.0
Maintained by
OpenSearch Foundation (AWS, others)
Typically used for
Log and metrics analytics, APM, full-text and semantic search, SIEM and security analytics, observability pipelines

The honest version

OpenSearch pros and cons

No technology is all upside. This is where OpenSearch earns its place and where it tends to cause trouble.

Where OpenSearch is strong

  • Apache 2.0 licenseFree of the Elastic License questions, which matters to procurement, to regulated environments, and to anyone shipping a product on top of the engine.
  • Tight AWS integrationAmazon OpenSearch Service wires to IAM, CloudWatch, VPC and S3 snapshots cleanly, and the managed control plane removes the worst of the upgrade work.
  • Index State Management built inISM policies move indices between hot, warm and cold tiers on a schedule, which keeps storage honest on log-heavy workloads without an external orchestrator.
  • Security plugin includedRole-based access control, LDAP and SAML integration, and document-level security come with the engine rather than as a paid add-on, which simplifies the stack for most teams.
  • Familiar surface for Elasticsearch usersEngineers who have run Elasticsearch find the APIs, the query DSL and the operational playbook largely intact. The migration curve is shallow in both directions.
  • Active anomaly detection and alertingThe anomaly detection, alerting and reporting plugins cover the pieces most observability teams would otherwise bolt on, which makes a single cluster a complete analytics layer rather than one component of many.

Where OpenSearch bites

  • JVM heap is still the first thing that goes wrongLong garbage collection pauses, fielddata pressure and circuit breakers firing on aggregations all trace back to heap sizing and the shape of the queries that touch it. The fork inherited this part of the parent engine unchanged.
  • Version drift from upstream ElasticsearchFour years on, APIs, mappings and plugins have diverged. Shared examples on the internet no longer always apply, and third-party tooling written for one engine occasionally breaks on the other.
  • ISM transitions can stall silentlyA policy that waits on an allocation rule or a shard count the cluster can no longer satisfy will retry forever without an obvious alarm, and the index stays on hot storage past its intended tier.
  • AWS blue-green deploys hide operational stateWhen the managed service upgrades or resizes a domain it does so through a blue-green process whose visibility is limited, and surprises during that window are hard to diagnose after the fact.
  • Observability is partly your problemCloudWatch metrics are useful and often not enough. Understanding shard distribution, hot threads and ingest backpressure usually needs direct API calls that CloudWatch does not surface.
  • Costs climb with log-heavy workloadsHot-tier instance hours, EBS throughput, UltraWarm and cold storage all stack up quickly, and it is common to find a domain costing more than the service it supports if nobody has reviewed the tiers in a year.

Sound familiar?

Common OpenSearch problems we fix

The OpenSearch problems that bring people to us, and what you have once each one is fixed.

cluster health status [RED], 8 unassigned shards

The cluster went red

A primary shard with no healthy copy, often after a disk watermark on a hot node, a stale replica after a network blip, or a corrupted index that cannot pass the recovery check.

Cluster green, shards assigned

ISM: policy retrying step 'attempt_rollover' for 12h

ISM is stuck on a transition

A rollover condition that no longer fires because the ingest rate dropped, or a warm tier whose allocation rule is unsatisfiable, so the policy loops without ever completing.

Policy conditions matched to real ingest

CircuitBreakingException: Data too large

Queries trip the parent circuit breaker

A dashboard asking for too many buckets at once, an aggregation on a text field, or a bulk request whose payload blew past the breaker. The breaker is correct; the shape of the work is wrong.

Queries and bulk sizes within heap limits

ClusterBlockException: cluster create-index blocked

The cluster is refusing new indices

A disk watermark that triggered a cluster-level block, or a limit on the number of shards per node that was reached after unmanaged index creation.

Headroom back, blocks released

shard copy marked as stale

A shard copy fell out of sync

A node that briefly lost quorum and came back behind, or a replica that drifted during an index that rapidly rolled over. The cluster isolates the stale copy, which can leave fewer replicas than planned.

Stale copies recovered, replication healthy

MasterNotDiscovered

A master node failover took too long

On the managed service, a blue-green deploy or an instance replacement that took longer than the timeout. For a few minutes clients saw errors that look unrelated to the actual cause.

Client retries aligned to real failover times

snapshot repository: 429 SlowDown from S3

Snapshots are being throttled by S3

A large manual snapshot that hits S3 request limits on a cold bucket, or a repository whose prefix is shared with other high-traffic data. Backups partially succeed and quietly fail retries.

Snapshot schedule and prefix tuned

ingest pipeline failed: UnknownHostException

Ingest pipelines are reaching outside the cluster

A geoip or enrich processor calling a service that is no longer reachable from the data nodes after a VPC change, so every document hits an error and goes to the dead-letter path or is dropped.

Pipelines self-contained or routed correctly

What we do

OpenSearch help, from rescue to steady running

Ring us for one red cluster, or let us own the search and logging stack.

Rescue
Red or yellow clusters, stuck ISM transitions, and runaway heap. We stabilize the domain first and write the incident up afterward.
Heap and query tuning
JVM settings, aggregations, mappings and shard sizing reviewed against real queries, with slow log numbers before and after.
ISM and tier design
Hot, warm and cold tiers, UltraWarm on AWS, and ISM policies set against real retention needs rather than copied from a template.
Migrations
Moving between Elasticsearch and OpenSearch, between self-hosted and Amazon OpenSearch Service, with parallel indices and dual writes so cutover is tested.
Cost review
Instance types, EBS throughput, tiers and snapshot repositories reviewed against query shape, with a bill before and after.
Managed Tech
Dashboards that watch shard counts, pending tasks, heap and GC rather than just CPU, with an engineer on call when a blue-green deploy lands badly.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running OpenSearch alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

OpenSearch questions

Something else? Call+1 737 727 7327

Why is our OpenSearch cluster yellow or red?

Yellow means at least one replica shard is unassigned, which is often tolerable on a small cluster and a warning on anything you care about. Red indicates a missing primary and therefore an availability problem until the shard is placed. The allocation explain API will tell you which shard and why, and the root cause is usually a disk watermark, a departed node, or an allocation rule that no longer resolves after a resize or an index template change.

Should we move from Elasticsearch to OpenSearch, or stay on Elastic?

That is a real decision with real tradeoffs. OpenSearch suits teams that want Apache 2.0 licensing, deep AWS integration and security without a paid tier. Elastic suits teams that want the newer vector and ML features, the official support channel, or the latest ILM and ES|QL work. We model the actual cost and feature gap against your workload rather than picking a side on principle.

How do we stop the JVM heap from running out on OpenSearch?

Start by measuring what is consuming it. The usual suspects are fielddata on analyzed text, aggregations producing enormous bucket counts, and bulk payloads that blow past the request limit. A heap sized above roughly 31 gigabytes stops paying for itself because compressed object pointers switch off, so throwing memory at the problem rarely helps. Most heap pressure clears up when a handful of problem queries get rewritten, keyword fields replace text fields for aggregations, and the shard count is brought down to a figure the master can actually schedule.

Can you take over an Amazon OpenSearch Service domain from a previous team?

Yes. We inventory the domain, the ISM policies, the index templates, the clients that write and read, and the IAM roles that back the snapshots. We rotate access, bring the configuration into Terraform or CloudFormation, and verify snapshots restore into a staging domain before changing anything.

How do we upgrade across a major OpenSearch version safely?

Not in a single step. We check deprecation logs on the current version, rehearse the upgrade on a snapshot restored into a sandbox, confirm every ingest pipeline and client library works on the target version, deprecate removed APIs in index templates, and then schedule the real upgrade in stages with a rollback plan ready.

How much does OpenSearch help cost?

The estimate is always free. Describe the domain, what it is indexing and what has gone wrong, and we send back a price before any changes land.

Free estimate

OpenSearch giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.