Running Apache Spark?

Spark jobs timing out or executors dying mid shuffle? We sort it out.

Most Spark jobs that land on our desk are doing roughly the right thing on a cluster that is roughly the wrong shape. A shuffle partition count stuck at 200 for a 500GB dataset, a join between a huge fact table and a tiny dimension that nobody told Spark to broadcast, a skewed key that puts 80 percent of the work on one executor. We read the Spark UI, measure the stage, and fix the actual bottleneck instead of the one that looks obvious.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

spark-job.log

still failing

What it is

Apache Spark at a glance

Apache Spark is a distributed processing engine for large datasets, started at UC Berkeley's AMPLab in 2009 and donated to the Apache Software Foundation in 2013. It unified batch processing, streaming, SQL and machine learning under a single engine, and became the default compute layer for data warehouses and data lakes over the following decade.

Modern Spark runs on YARN, Kubernetes, Databricks, EMR, Dataproc and standalone clusters. Most new code uses the DataFrame and Spark SQL APIs rather than raw RDDs, with Delta Lake, Apache Iceberg or Hudi providing the storage layer. Jobs are typically written in Scala, Python through PySpark, or plain SQL.

Type
Distributed data processing engine
First released
2014 Apache top level project
License
Apache 2.0, open source
Maintained by
Apache Software Foundation and a wide contributor base
Typically used for
ETL at scale, data warehouses, ML feature pipelines, streaming aggregations

The honest version

Apache Spark pros and cons

No technology is all upside. This is where Apache Spark earns its place and where it tends to cause trouble.

Where Apache Spark is strong

  • Scales from a laptop to thousands of coresThe same DataFrame code runs on a single JVM for testing and across a managed cluster for production. Few engines have that range in practice.
  • Catalyst and Tungsten do real optimisationPredicate pushdown, projection pruning, cost based join reordering and code generation mean naive SQL often performs within reach of hand tuned code.
  • Adaptive query execution helps bad plansAQE coalesces small partitions, switches join strategies based on actual sizes, and handles skew automatically on hot keys. It is not a silver bullet, but it hides a lot of past pain.
  • First class SQL on open formatsParquet, Delta, Iceberg and Hudi are all natively supported. The same query engine serves notebooks, dashboards and production jobs.
  • Managed offerings mean the cluster is not your problemDatabricks, EMR Serverless and Dataproc remove the operational layer for teams that would rather pay than run a cluster themselves.

Where Apache Spark bites

  • Memory tuning is still a dark artExecutor memory, overhead, off heap, broadcast threshold and shuffle buffers all interact. Getting them right is not well documented for your specific workload and often takes real measurement.
  • Shuffle is usually the bottleneckLarge joins, wide aggregations and repartitions move huge amounts of data across the network. Most Spark performance work is really shuffle performance work.
  • Data skew wrecks otherwise fine jobsA handful of keys with much more data than the rest can turn a 10 minute job into a 4 hour one. AQE helps; it does not catch every pattern.
  • PySpark adds a serialisation taxPython UDFs cross the JVM to Python boundary on every row. Pandas UDFs and arrow help; naive PySpark code regularly runs five times slower than the Scala equivalent.
  • The dependency and version matrix is painfulHadoop, Scala and Java versions have to agree, and a Databricks runtime pin can quietly diverge from vanilla Spark. Classpath conflicts land often.
  • Costs compound on managed servicesDBUs, EMR node hours and S3 request charges each look small and add up to large monthly bills, especially when jobs are left on default settings.

Sound familiar?

Common Apache Spark problems we fix

The Apache Spark problems that bring people to us, and what you have once each one is fixed.

Container killed by YARN for exceeding memory limits

Executors die on shuffle or GC

Memory overhead set too low for the real working set, a single partition that will not fit, or a UDF pulling the whole row into Python. Raising memory helps once; fixing the plan helps for good.

Executors stable under production load

FetchFailedException: Connection reset by peer

Shuffle fetches are failing intermittently

Executor death cascading into fetch failures, network hiccups between nodes, or external shuffle service misconfiguration. The symptom is downstream of a different problem.

Shuffle stages complete without retries

Skewed partition 18x larger than median

Data skew is destroying the stage

A hot key, a null value treated as a group, or a join with a many to many fan out. AQE skew handling, salted joins or a repartition on a better key each work in different situations.

Work distributed evenly across executors

spark.sql.autoBroadcastJoinThreshold exceeded

Join strategy is wrong for the actual data

A SortMergeJoin where a broadcast would fit, or a broadcast attempted on a table that is too big and crashes the driver. Stats, hints and the plan all need checking.

Join strategies matched to table sizes

No space left on device at /mnt/local/shuffle

Local shuffle storage has filled up

A huge shuffle that spilled to disk and did not get cleaned up, or node local SSDs sized for a smaller workload. Either grow the storage or shrink what gets written.

Spill storage sized for the real shuffle

Task of very large size

Partitions are too big to be tasks

A read with too few partitions, or a repartition count stuck on the default 200 when the data needs thousands. Each task tries to hold too much and the executors buckle.

Partitions sized to the cluster

AnalysisException: Cannot resolve column name

Schema has drifted under the job

An upstream producer renamed a column, a Delta Lake table underwent a schema evolution the consumer did not expect, or a case sensitivity mismatch between the catalog and the query. Jobs fail cleanly rather than guessing.

Schema contract honoured end to end

What we do

Spark help, from one failing job to the whole platform

Reach us for a single OOM, or let us run the Spark platform on retainer.

Rescue
A job that will not finish, a cluster that keeps dying, or an inherited notebook nobody can read. We stabilise the pipeline and write down what changed.
Performance tuning
Shuffle sizing, broadcast thresholds, AQE configuration and executor memory tuned against real stage traces from your production runs.
Skew and join rewrites
Hot keys salted, many to many joins restructured, and skew handling verified with before and after measurements rather than claims.
PySpark to Scala and back
Hot paths rewritten in Scala or Spark SQL where Python UDF overhead is the real cost, with a plan for which code stays Pythonic for productivity.
Delta, Iceberg and lakehouse layout
Table partition layout, Z ordering, compaction and vacuum policies tuned so read and write workloads do not fight each other.
Managed Tech
An ongoing engagement covering job observability, cost reviews, cluster upgrades and a Spark engineer on call when an overnight pipeline misses its SLA.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Apache Spark alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Apache Spark questions

Something else? Call+1 737 727 7327

Why is our Spark job so slow?

In almost every case the real cost is in shuffle. Most tuning conversations end up being about partition count, join strategy, data skew and where unnecessary wide operations sneaked in. We look at the Spark UI for actual stage timings before touching any configuration flag.

Should we move from EMR to Databricks, or the other way?

Honest answer: it depends on who is operating it and what the price per useful hour of compute actually looks like for your workloads. Databricks is more productive out of the box; EMR Serverless is cheaper when workloads are already optimised. We benchmark on your real jobs before recommending.

Can you help with PySpark that is too slow?

Yes. We measure where the time is going, replace Python UDFs with Spark SQL or Pandas UDFs where they are the bottleneck, and rewrite hot paths in Scala only when the difference justifies the maintenance cost. We do not rewrite for the sake of it.

Why does our Spark job work on small data and fail on production volumes?

Usually a partition count, broadcast threshold or driver memory that was fine at 10GB and falls over at 500GB. Skew that did not exist at small scale, joins whose strategy flipped, and shuffle stages whose spill no longer fits on local disk. Each one needs its own fix.

Can you take over maintenance of an existing Spark pipeline?

Yes. We inventory the jobs, their schedules and their dependencies, document ownership, and bring the whole surface under observability before changing anything. Only then do we tune.

How much does Apache Spark help cost?

Estimates are provided at no charge. Share the Spark UI link, a job description or a stage trace and we come back with a figure before the work begins.

Free estimate

Spark giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.