Scheduled runs are stuck in Late
A work pool exists, the deployment is pointed at it, and no worker process is polling it. The dashboard shows the run as Late indefinitely until a worker picks up the pool.
Workers polling, runs starting
On Prefect?
Prefect flows look like ordinary Python until the deployment, work pool and infrastructure block have to agree on which runtime will actually pick up the next run. When they disagree, scheduled runs sit in Late indefinitely and the UI gives you very few hints. We trace it to the specific config and fix it.
What it is
Prefect is a Python native workflow orchestrator from Prefect Technologies, originally released in 2018 and significantly rewritten as Prefect 2 in 2022. It lets you add decorators to ordinary Python functions and get retries, observability, scheduling and distributed execution without rewriting the function.
The platform centres on flows, which are Python functions decorated with @flow, and tasks, which are smaller units of work inside them. Deployments associate a flow with a schedule, a work pool and an infrastructure configuration. Prefect Cloud, the managed hosted control plane, handles scheduling, UI and observability, while workers inside your environment pick up and execute runs.
The honest version
No technology is all upside. This is where Prefect earns its place and where it tends to cause trouble.
Sound familiar?
The Prefect problems that bring people to us, and what you have once each one is fixed.
A work pool exists, the deployment is pointed at it, and no worker process is polling it. The dashboard shows the run as Late indefinitely until a worker picks up the pool.
Workers polling, runs starting
An IAM role missing a permission, a Kubernetes service account that cannot create jobs, or a container image that does not exist. The worker logs are where the real error lives.
Infrastructure permissioned correctly
An import error, a missing environment variable, or a configuration call that fails at module load time. Prefect never gets to execute the @flow because the module cannot be loaded.
Modules load before Prefect runs them
A deployment was created without a cron or interval schedule, so it is only runnable on demand. Easy to miss when porting from a cron based system that used to trigger externally.
Schedules explicit and visible
A task is trying to pass a non picklable object, such as a database session or a client with a lock, between tasks. The fix is to construct it inside the task rather than passing it in.
Tasks exchange serialisable values
A missing env var, a wrong entrypoint, or an image that starts and crashes within seconds. The ECS task logs end before the worker even registers the run.
ECS tasks live long enough to work
A utility module calls get_run_logger or similar outside an active flow context, often during a unit test. Guarding the call or passing the logger explicitly resolves it.
Context respected across modules
A worker or a sync script hammering the API, often because of a tight polling loop set up during debugging. We back off the polling and scope API tokens to the right surface.
API traffic inside the plan
What we do
Call us for one Late deployment, or let us design the whole orchestration setup.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
Almost always because no worker is polling the work pool the deployment points at. It can also be a worker that is polling a different pool, a work pool with a type mismatch against the worker, or a worker that crashed and was never restarted. The UI shows pool health; the worker logs show the polling behaviour. We confirm the pool and worker are genuinely paired and the worker process is alive.
Prefect fits best when the workflow is a lot of Python glue, dynamic branches and existing functions you want to decorate rather than rewrite. Dagster fits best when the pipeline is mostly about producing data assets with clear lineage and partitioning. Both are credible Airflow alternatives, and we can map your real pipelines onto each before suggesting one.
Smaller than 1 to 2 but still real. Blocks, deployments and worker semantics changed, and some APIs were tightened. We run both versions in parallel against a shared database, port flows in batches, and only cut over once the Prefect 3 version has run a full reporting cycle without drift.
Prefect serialises task outputs so they can be persisted and passed between retries. Objects that wrap threads, locks, SQLAlchemy sessions or open sockets cannot be pickled. The fix is to construct those objects inside the task that uses them and return simple data, not to try to serialise a live connection.
Yes. We export the DAGs that matter, rewrite them as Prefect flows that call the same business logic, run both platforms in parallel against a shared source of truth, and only retire the Airflow DAG once outputs have matched across a representative window.
Estimates are free. Send your flow code and a recent Late run, and we come back with a price and a short plan before anything gets changed.
Free estimate
Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: