On Prefect?

Prefect deployments stuck in Late or work pools idle? We sort it out.

Prefect flows look like ordinary Python until the deployment, work pool and infrastructure block have to agree on which runtime will actually pick up the next run. When they disagree, scheduled runs sit in Late indefinitely and the UI gives you very few hints. We trace it to the specific config and fix it.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

prefect.log

still broken

What it is

Prefect at a glance

Prefect is a Python native workflow orchestrator from Prefect Technologies, originally released in 2018 and significantly rewritten as Prefect 2 in 2022. It lets you add decorators to ordinary Python functions and get retries, observability, scheduling and distributed execution without rewriting the function.

The platform centres on flows, which are Python functions decorated with @flow, and tasks, which are smaller units of work inside them. Deployments associate a flow with a schedule, a work pool and an infrastructure configuration. Prefect Cloud, the managed hosted control plane, handles scheduling, UI and observability, while workers inside your environment pick up and execute runs.

Type
Workflow orchestrator, Python native
First released
2018; Prefect 2 in 2022; Prefect 3 in 2024
License
Apache 2.0 open source; Prefect Cloud is a paid SaaS
Maintained by
Prefect Technologies, Inc.
Typically used for
Python data pipelines, batch ETL, ML workflows, scheduled Python jobs

The honest version

Prefect pros and cons

No technology is all upside. This is where Prefect earns its place and where it tends to cause trouble.

Where Prefect is strong

  • Ordinary Python stays ordinaryA @flow decorator on a function turns it into a scheduled orchestrated workflow without restructuring the code, which makes adoption much lower friction than Airflow.
  • Dynamic control flowLoops, conditionals and parameter driven branches work the way Python does, rather than needing an operator for every possible shape of DAG.
  • Good retry and caching primitivesRetries, caching keys, task tags and timeouts are decorator arguments rather than separate concepts, which keeps the orchestration readable in the function itself.
  • Hybrid execution with Prefect CloudThe hosted control plane handles scheduling and UI, workers run inside your own network, so data and code never leave your infrastructure even with the managed offering.
  • Decent async and concurrency supportAsync flows, task runners and concurrency limits are built in, so IO heavy pipelines can parallelise without a separate worker framework.
  • Reasonable free tier on Prefect CloudThe free tier covers small teams and side projects, which makes it easy to prove out before committing to a paid plan.

Where Prefect bites

  • Deployment, work pool, worker, infrastructure split confuses new usersWhere the schedule lives, what runs the flow, and which infrastructure block applies are four different concepts, and getting one wrong leaves runs sitting in Late with no obvious cause.
  • Big breaking changes between major versionsPrefect 1 to 2 to 3 each rewrote significant surface area, and teams that skipped versions can face a substantial port rather than a smooth upgrade.
  • Smaller ecosystem than AirflowCollections cover the mainstream integrations, but niche source systems often mean writing your own Python wrapper rather than reaching for an off the shelf operator.
  • Error messages can be thinA late flow run, a worker that is not polling or an infrastructure block with the wrong IAM can produce almost identical surface symptoms, and tracing the root cause often means reading worker logs rather than the UI.
  • Prefect Cloud pricing shiftsPricing has changed meaningfully across the product's history, and teams rely on blog posts that no longer describe the current plan when budgeting.
  • Pickle based serialisation surprisesObjects passed between tasks have to be picklable, and classes holding threads, sockets or SQLAlchemy sessions produce cryptic serialisation errors.

Sound familiar?

Common Prefect problems we fix

The Prefect problems that bring people to us, and what you have once each one is fixed.

Work pool has no polling workers

Scheduled runs are stuck in Late

A work pool exists, the deployment is pointed at it, and no worker process is polling it. The dashboard shows the run as Late indefinitely until a worker picks up the pool.

Workers polling, runs starting

Infrastructure type failed to submit

Infrastructure block fails to launch the run

An IAM role missing a permission, a Kubernetes service account that cannot create jobs, or a container image that does not exist. The worker logs are where the real error lives.

Infrastructure permissioned correctly

ScriptError: exception while running flows/etl.py

Flow script errors before Prefect starts it

An import error, a missing environment variable, or a configuration call that fails at module load time. Prefect never gets to execute the @flow because the module cannot be loaded.

Modules load before Prefect runs them

Deployment has no schedule set

Scheduled deployment is not scheduled

A deployment was created without a cron or interval schedule, so it is only runnable on demand. Easy to miss when porting from a cron based system that used to trigger externally.

Schedules explicit and visible

cannot pickle '_thread.lock' object

Task serialisation fails

A task is trying to pass a non picklable object, such as a database session or a client with a lock, between tasks. The fix is to construct it inside the task rather than passing it in.

Tasks exchange serialisable values

ECS task stopped: Essential container exited

ECS infrastructure block exits immediately

A missing env var, a wrong entrypoint, or an image that starts and crashes within seconds. The ECS task logs end before the worker even registers the run.

ECS tasks live long enough to work

MissingContextError: no active flow run

Prefect calls made outside a flow context

A utility module calls get_run_logger or similar outside an active flow context, often during a unit test. Guarding the call or passing the logger explicitly resolves it.

Context respected across modules

429 Rate limit exceeded on Prefect Cloud API

Prefect Cloud API is rate limiting

A worker or a sync script hammering the API, often because of a tight polling loop set up during debugging. We back off the polling and scope API tokens to the right surface.

API traffic inside the plan

What we do

Prefect help, from rescue to platform design

Call us for one Late deployment, or let us design the whole orchestration setup.

Rescue
Flows stuck in Late, workers that will not poll, or infrastructure blocks that refuse to launch. We find the specific cause and get the next run through.
Deployment and work pool design
The right split of work pools, workers and infrastructure for your environments, with schedules that match the SLA each flow actually has.
Infrastructure block setup
Kubernetes, ECS, Docker and process based infrastructure configured with the right IAM, secrets and image handling so runs start reliably.
Migrations
Moves from Airflow, cron or a previous Prefect version done in a measured pass, with the old system running alongside until the new one is trusted.
Prefect Cloud setup
Workspaces, blocks, variables and workers wired into pull requests and deploy pipelines so flows are managed as code rather than clicked together.
Managed Tech
Ongoing reviews of flow success rate, late run alerts, infrastructure health and cost, plus a line to call when a nightly pipeline fails at 3am.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Prefect alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Prefect questions

Something else? Call+1 737 727 7327

Why are our Prefect deployments stuck in Late?

Almost always because no worker is polling the work pool the deployment points at. It can also be a worker that is polling a different pool, a work pool with a type mismatch against the worker, or a worker that crashed and was never restarted. The UI shows pool health; the worker logs show the polling behaviour. We confirm the pool and worker are genuinely paired and the worker process is alive.

Should we pick Prefect or Dagster for a new project?

Prefect fits best when the workflow is a lot of Python glue, dynamic branches and existing functions you want to decorate rather than rewrite. Dagster fits best when the pipeline is mostly about producing data assets with clear lineage and partitioning. Both are credible Airflow alternatives, and we can map your real pipelines onto each before suggesting one.

How painful is the Prefect 2 to 3 migration?

Smaller than 1 to 2 but still real. Blocks, deployments and worker semantics changed, and some APIs were tightened. We run both versions in parallel against a shared database, port flows in batches, and only cut over once the Prefect 3 version has run a full reporting cycle without drift.

Why does our flow fail with a pickle error on task output?

Prefect serialises task outputs so they can be persisted and passed between retries. Objects that wrap threads, locks, SQLAlchemy sessions or open sockets cannot be pickled. The fix is to construct those objects inside the task that uses them and return simple data, not to try to serialise a live connection.

Can you help us move off Airflow to Prefect?

Yes. We export the DAGs that matter, rewrite them as Prefect flows that call the same business logic, run both platforms in parallel against a shared source of truth, and only retire the Airflow DAG once outputs have matched across a representative window.

How much does Prefect help cost?

Estimates are free. Send your flow code and a recent Late run, and we come back with a price and a short plan before anything gets changed.

Free estimate

Prefect leaving runs Late?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.