Running Elasticsearch?

Elasticsearch cluster yellow, red, or crawling? We unstick it.

The cluster sat on green for a year, then one node ran out of disk at 3am, shards went unassigned, and nothing has been quite right since. We open the allocation explain output, read the heap and GC numbers from the slow logs, and bring the cluster back to a state that lets you sleep.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

elasticsearch.log

still broken

What it is

Elasticsearch at a glance

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It stores JSON documents in indices, splits them into shards across nodes, and serves near-real-time full-text search, aggregations and vector queries over them. It is the engine behind most of the log, metrics and APM pipelines people call the Elastic Stack, alongside Kibana for the UI, Logstash and Beats for ingest, and Fleet for agent management.

Shay Banon released the first version in 2010 and the company behind it, Elastic N.V., went public in 2018. The project has moved licenses more than once: the current Elasticsearch is dual-licensed under the Server Side Public License and the Elastic License, which led AWS to fork it as OpenSearch in 2021. Most trouble in production comes from the same handful of places, including JVM heap pressure, shard counts that have crept out of hand, mapping explosions on dynamic fields, and lifecycle policies that were set once and never revisited.

Type
Distributed search and analytics engine
First released
2010
License
Dual: SSPL and Elastic License v2
Maintained by
Elastic N.V.
Typically used for
Full-text search, log and metrics analytics, APM, vector and semantic search, SIEM

The honest version

Elasticsearch pros and cons

No technology is all upside. This is where Elasticsearch earns its place and where it tends to cause trouble.

Where Elasticsearch is strong

  • Search that scales without rewriting the schemaFull-text queries with ranking, filters and aggregations across billions of documents are a standard workload rather than a research project.
  • Near-real-time ingest and queryDocuments are searchable within a second of being indexed, which fits log pipelines and user-facing search in the same cluster.
  • A wide query languageQuery DSL covers full text, structured filters, geo, aggregations and now vector search, so one engine handles cases that would otherwise need several.
  • Index lifecycle managementILM moves indices through hot, warm, cold and frozen tiers automatically, which keeps storage costs sane for log-heavy workloads.
  • A big, honest ecosystemLogstash, Beats, Fleet, Kibana, APM and the many community clients cover ingest and visualization without hand-written glue.
  • Snapshots to object storageBackups to S3, Azure Blob or GCS are a first-class feature rather than an afterthought, and restores are routine when they have been tested.

Where Elasticsearch bites

  • The JVM heap is a sharp edgeLong GC pauses, field data pressure and parent circuit breakers tripping all come back to heap sizing and query shape. Tuning the JVM is still a required skill.
  • Shard counts creep upwardA cluster with ten thousand tiny shards spends half its time on cluster-state housekeeping, and everyone is afraid to reindex because downtime is painful.
  • Mapping explosions are easy to causeDynamic templates turned on against JSON from an unknown source can create thousands of fields in a day, which pushes the cluster state past safe limits.
  • Upgrades across major versions need planningDeprecations between 6.x, 7.x, 8.x and 9.x hit mappings, security settings, pipelines and client libraries, so a rushed upgrade will usually leave something limping.
  • Licensing has shifted and may shift againThe move to SSPL and the Elastic License, and the resulting OpenSearch fork, mean procurement and compliance now have questions they did not have five years ago.
  • Costs climb with log-heavy workloadsHot storage, dedicated masters, enough heap to run aggregations and frozen tier retrieval all add up fast, and it is common to see a cluster quietly costing more than the application it serves.

Sound familiar?

Common Elasticsearch problems we fix

The Elasticsearch problems that bring people to us, and what you have once each one is fixed.

cluster health status [RED], 14 unassigned shards

The cluster went red

A primary shard with no healthy copy, usually after a node failure, a disk watermark, or a corrupted index. The allocation explain API shows which shard and why; the fix depends on whether a replica exists or a snapshot restore is needed.

Cluster green, shards assigned

CircuitBreakingException: [parent] Data too large

Queries trip the circuit breaker

An aggregation building a large structure, field data on a text field, or a dashboard asking for too many buckets at once. The breaker is doing its job, the shape of the query is wrong.

Queries and field usage within heap limits

high disk watermark exceeded

A node hit the disk watermark and stopped accepting shards

Logs or old indices filled the data disk past the low or high watermark, so Elasticsearch marks the node unavailable for allocation and may set indices to read-only.

Headroom back, lifecycle policy watching it

pending_tasks: 1,402

The master is overloaded

A cluster with far too many shards or very chatty mappings is spending all its time on cluster-state updates, and ordinary operations queue behind the master.

Shard count sane, master calm

max_primary_shard_size not reached

Index lifecycle rollover never fires

ILM waiting on a size threshold that a low-traffic stream will never hit, so indices grow unbounded and rollover does not happen.

Lifecycle set against real ingest rate

bulk rejections rising

Logstash or Beats keep getting bulk rejections

Ingest nodes saturated, the bulk queue filling, or coordinating nodes undersized for the rate. Clients retry and the problem compounds.

Ingest keeps up, back-pressure handled

MapperParsingException: timestamp

Documents are being rejected on ingest

A new source sending dates in a format the mapping does not accept, or a field that was once numeric now arriving as a string, with no dead-letter path.

Pipeline tolerant, mapping explicit

cluster_block_exception: read-only / allow delete

An index is read-only after the disk filled

Elasticsearch set the index block automatically when the disk hit the flood-stage watermark, and nobody cleared the block after freeing space.

Index writable, watermarks monitored

What we do

Elasticsearch help, from rescue to steady running

Call us for one red cluster, or hand us the whole search and logging stack.

Rescue
Red or stuck clusters, unassigned shards, nodes refusing to join. We stabilize the cluster first and write up what happened second.
Heap and query tuning
JVM settings, aggregations, mappings and shard sizing reviewed against the real queries, with slow log numbers before and after.
Upgrades and reindexing
Major version upgrades, index-by-index reindexing, and migrations between on-prem clusters, Elastic Cloud and OpenSearch, rehearsed on a copy first.
Ingest and ILM
Logstash, Beats and Fleet pipelines that drop or stall, with lifecycle policies set for real retention and real disk.
Cost review
Hot, warm, cold and frozen tiers set against how the data is actually queried. Most log clusters have spare money hiding on the hot tier.
Managed Tech
Monitoring that watches heap, GC and shard counts rather than just up or down, with someone on call when a node starts dropping replicas.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Elasticsearch alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Elasticsearch questions

Something else? Call+1 737 727 7327

Why is our Elasticsearch cluster yellow or red?

Yellow means at least one replica shard is unassigned, which is often harmless on a single-node setup but a warning sign in production. Red means a primary is unassigned, which is a data-availability problem until it is resolved. The allocation explain API will tell you exactly which shard and why, and the answer is usually a disk watermark, a failed node, or an index whose allocation rules can no longer be satisfied.

How do we stop the JVM heap from running out?

By understanding what is actually using it. Field data on text fields, large aggregations, parent-child joins and oversized bulk requests are the usual culprits. Heap bigger than 31 or 32 GB is counterproductive because of compressed object pointers. Most heap problems we see are fixed by rewriting a handful of queries, replacing text with keyword fields for aggregations, and setting a sensible shard count rather than by adding more memory.

Can you take over an Elasticsearch cluster from a previous team?

Yes. We review the node roles, mappings, index lifecycle policies, snapshots and clients, document which applications depend on which indices, and bring the configuration back into version control before changing anything. Takeovers are a common piece of this work.

Should we move from Elasticsearch to OpenSearch, or stay put?

Only when there is a reason. OpenSearch fits workloads that want an Apache 2.0 license and tight AWS integration. Elastic suits teams that want the newer vector search features, the licensed security and ML add-ons, or support from the company behind the engine. We will model the real cost and feature gap for your cluster rather than pick a side on principle.

How do we upgrade across a major Elasticsearch version safely?

Not on a Friday afternoon. We check deprecation logs on the current cluster, rehearse the upgrade on a snapshot restored into a sandbox, confirm every client library and ingest pipeline works on the target version, and then schedule the real one in stages with a rollback plan.

How much does Elasticsearch help cost?

Estimates are always free. Describe the cluster, what it is indexing and what is going wrong, and you get a price before any work begins.

Free estimate

Elasticsearch cluster worrying you?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.