Running Apache Airflow?

Airflow DAGs stuck or scheduler lagging? We unstick it.

Airflow is the kind of system that works beautifully until it does not, and when it does not the whole data platform downstream stops. We walk the DAG parse times, the scheduler loops, the executor queue and the worker pods until the real bottleneck is on screen, and then we fix it without rewriting all your pipelines.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

airflow-scheduler.log

still broken

What it is

Apache Airflow at a glance

Apache Airflow is an open-source workflow orchestrator originally built at Airbnb in 2014 and donated to the Apache Software Foundation. It expresses workflows as Directed Acyclic Graphs of tasks written in Python, with a scheduler, a metadata database, a web UI and an executor (Local, Celery, or Kubernetes) that runs the tasks.

It is now the default choice for ETL and data pipeline orchestration in mid-size and large data teams, with managed offerings on AWS (MWAA), Google Cloud (Cloud Composer) and Astronomer. DAGs sit in a folder, are parsed by the scheduler on an interval, serialized into the metadata database, and then executed by workers. Nearly every production incident we see comes from one of those four moving parts being out of balance with the others.

Type
Workflow orchestrator for data pipelines
First released
2015 as Airbnb open source, 2016 as Apache incubator
License
Apache 2.0
Maintained by
Apache Software Foundation
Typically used for
ETL pipelines, data warehouse loads, ML training workflows, report generation

The honest version

Apache Airflow pros and cons

No technology is all upside. This is where Apache Airflow earns its place and where it tends to cause trouble.

Where Apache Airflow is strong

  • DAGs as Python codePipelines are real code, in version control, with tests and reviews, which is the single biggest reason teams pick it over point-and-click orchestrators.
  • Large operator catalogHooks and operators exist for most warehouses, cloud services and SaaS APIs you are likely to touch, so new pipelines rarely start from scratch.
  • Managed offerings reduce the worst opsMWAA, Cloud Composer and Astronomer take the scheduler, database and worker fleet off your plate, which removes a lot of the sharpest edges.
  • Granular retries and SLAsPer-task retries, exponential backoff, SLAs, trigger rules and sensor patterns cover most of the orchestration needs real pipelines have.
  • Observability on real runsTask duration histories, Gantt views, logs per task instance and run comparison are all in the web UI, which makes debugging a slow pipeline much faster than in a bespoke orchestrator.

Where Apache Airflow bites

  • Scheduler tuning is a craftDAG parse intervals, min file process interval, max_threads, parsing processes and pool sizes all interact. Getting them wrong manifests as 'scheduler lag' and queued tasks that never run.
  • DAG parse time is a hidden taxEvery DAG file is reparsed on an interval. Slow imports, API calls at the top level, or heavy dynamic DAG generation make the scheduler fall behind for the whole system.
  • Metadata database is a single point of pressureA large Airflow runs most of its moving parts through Postgres or MySQL. Connection leaks, lock contention and an under-sized database are a frequent cause of outages.
  • Upgrades across minor versions require careProvider packages split out, serialization formats changed, and Python compatibility windows closed. In-place upgrades past 2.3 need a plan rather than a package bump.
  • Kubernetes executor amplifies failure modesPod launches that race against quotas, PVCs that fail to bind, and image pulls that time out all look like Airflow problems when they are really cluster problems.
  • Not a good fit for streamingAirflow is batch. It will schedule frequent runs but it is not the right tool for event-driven or sub-minute streaming workloads, and teams that stretch it into that regret the choice.

Sound familiar?

Common Apache Airflow problems we fix

The Apache Airflow problems that bring people to us, and what you have once each one is fixed.

Task stuck in queued 1h 42m

Tasks never leave the queue

The executor cannot claim the task: pool exhausted, worker fleet undersized, Celery queue misnamed, or the Kubernetes namespace out of quota. The UI shows queued, the logs show nothing.

Executor and pool balanced to real demand

DagFileProcessorTimeout 50s

DAG parsing times out

A DAG file that makes API calls at the top level, imports heavy modules, or generates hundreds of DAGs dynamically, pushing parse time past the scheduler's limit.

Parse costs lifted out of top-level code

scheduler heartbeat delay 97s

Scheduler is falling behind

Too many DAGs for one scheduler, parse intervals set too aggressive, or the metadata database running out of IOPS. Everything downstream waits.

Scheduler sized and configured for your fleet

ImportError after upgrade

DAGs broken after an Airflow upgrade

Provider packages split out, module paths moved, and old import lines fail. The web UI turns red with 'Broken DAG' errors across the whole fleet.

Imports migrated, DAGs green again

DagRun already exists

Backfills clash with the scheduler

A manual backfill overlapping the normal schedule, duplicate DAG runs on the same execution date, and tasks that will not start because of the clash.

Backfill plan that does not collide

KubernetesExecutor pods forbidden

Kubernetes executor cannot launch pods

A ServiceAccount missing permissions, a namespace without ResourceQuota headroom, or a PodTemplate that references a nonexistent image pull secret.

RBAC and quotas set correctly

Zombie task detected

Tasks go zombie on workers

A worker that crashed mid-task and did not update the heartbeat, leaving the task record thinking the task is still alive long after the worker is gone.

Heartbeat and worker health lined up

pool default_pool exhausted

Pool configuration bottlenecks everything

One pool handling all traffic, with far fewer slots than concurrent tasks, so a heavy daily job starves every other DAG in the system.

Pools split by workload class

What we do

Airflow help, from rescue to scale

Hire us for one broken DAG, or let us run the whole Airflow platform.

Rescue
Scheduler lagging, tasks queued forever, or the whole fleet showing broken DAGs after an upgrade. We get the pipelines moving first and then write up the cause.
Scheduler and executor tuning
Parse intervals, pool sizes, parallelism, worker counts and metadata database settings sized against your real DAG fleet rather than defaults.
DAG design review
Idempotence, retries, SLAs, backfill behavior and testability reviewed for your most important DAGs, with concrete rewrites where it matters.
Upgrades
Minor-version upgrades of Airflow and its provider packages rehearsed on a staging instance, with every DAG verified before cutover.
Managed migrations
Moving onto or off Airflow, between self-hosted and MWAA/Cloud Composer/Astronomer, or splitting one Airflow into tenants per team.
Managed Tech
Ongoing coverage: scheduler health dashboards, DAG parse budgets, pool monitoring and an on-call responder when a critical DAG runs late.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Apache Airflow alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Apache Airflow questions

Something else? Call+1 737 727 7327

Why are our Airflow tasks stuck in queued forever?

Queued means the scheduler has decided a task should run but the executor has not picked it up. The usual causes are a pool with no free slots, a Celery queue name mismatch between scheduler and worker, a Kubernetes executor with insufficient RBAC or quota, or a worker fleet simply too small for the DAG fleet. The scheduler log will say which if you pipe it somewhere readable.

Our Airflow scheduler is lagging, what should we tune?

Start with DAG parse cost: grep for top-level API calls and heavy imports, and factor them out. Then look at scheduler config (parsing_processes, min_file_process_interval, max_threads), check the metadata database for IOPS saturation, and confirm the number of schedulers matches the DAG count. In practice one of those four moves fixes most lag cases.

We upgraded Airflow and all our DAGs are broken, can you help?

Yes, this is a very common engagement. The 2.x line split providers out of the core package and renamed several import paths, which breaks older DAGs on load. We work through the import errors by provider, patch the DAGs in a branch, and run them against a staging Airflow until the UI is clean, then cut over the production scheduler.

Should we use MWAA, Cloud Composer, Astronomer, or self-host?

If data engineering capacity is thin, a managed offering will almost always pay for itself by removing scheduler and database operations work. If you have a platform team and strict network requirements, self-hosting on Kubernetes gives more control. We lay out the real costs including on-call time before recommending either direction.

How do we avoid Airflow outages during a backfill?

Use a separate pool for backfills so they cannot starve production DAGs, cap parallelism per DAG, and avoid overlapping execution dates with the normal schedule. For large backfills we also spin up extra workers temporarily rather than letting the normal fleet carry the whole load.

How much does Airflow help cost?

Quotes are free. Describe the executor, the version and what is going wrong and we come back with a scoped price before any work begins.

Free estimate

Airflow giving you trouble?

Tell us the executor, the version and what is misbehaving and we reply with a plan and a quote.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.