Deploying on Modal?

Modal cold starts or GPU quotas biting? We fix it.

Modal is the fastest way to turn a Python function into a serverless endpoint with a GPU behind it, and the places where that story cracks are specific: cold starts that blow through your latency budget, App.lookup calls that race against deployments, GPU quotas that only announce themselves under load. We tune the images, the concurrency and the lifecycle so the workload runs the way the demo promised.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

modal.log

still failing

What it is

Modal at a glance

Modal is a serverless Python platform aimed at machine learning, data processing and agent workloads. You decorate functions with @app.function, specify an image, CPU, memory and optional GPU, and Modal takes care of running the function on demand with whatever concurrency you ask for.

The product is used heavily for inference endpoints, fine-tuning jobs, batch ETL, scraping pipelines and background work behind agent systems. GPU options range from T4 up through A100 80GB and H100. Pricing is per-second for compute with a free monthly allowance; you only pay while functions are actively running. The developer experience is explicitly about keeping everything in Python, including dependencies and infrastructure definitions.

Type
Serverless Python compute platform with GPU support
First released
2022
License
Proprietary SaaS, pay-per-second usage
Maintained by
Modal Labs
Typically used for
ML inference, fine-tuning, batch jobs, serverless agents, scheduled tasks

The honest version

Modal pros and cons

No technology is all upside. This is where Modal earns its place and where it tends to cause trouble.

Where Modal is strong

  • GPUs are first classT4, L4, A10, A100 and H100 all appear as a one-line decorator argument, with real capacity behind them rather than the empty queues that plague most generic clouds.
  • Python-native infrastructureDockerfiles, Terraform and Helm are not required. The image, mounts, secrets, volumes and schedules are all defined in the same Python file as the function.
  • Fast iterationmodal run and modal serve push a code change and let you call it over the internet within seconds, which makes the feedback loop closer to local development than to Kubernetes.
  • Reasonable cold starts for MLImage caching and GPU pre-warming keep cold starts down relative to generic FaaS, especially on common model weights that live on network volumes.
  • Scheduled and background workCron, retries, queues and web endpoints are built in, so a batch job, a webhook handler and a GPU inference endpoint share one mental model.
  • Transparent pricingYou pay for GPU-seconds and CPU-seconds while the function runs, not for reserved instances or idle time. Bills tend to match intuition.

Where Modal bites

  • Cold starts still happenFirst invocations after idle, large image pulls and model weight loads add seconds that matter for interactive endpoints. Keep-warm settings cost money, so there is a real trade-off.
  • GPU quotas can surprise youCapacity for A100 80GB and H100 is not infinite. A workload that scaled fine during the day can hit a regional quota on a Saturday evening and start queueing.
  • Debugging is remote by natureA traceback that reproduces locally does not always reproduce in the container, and the shell experience, while present, is less immediate than SSH into a VM.
  • Vendor coupling is realThe decorator-based API is pleasant and specific to Modal. Moving a non-trivial workload off Modal means rebuilding the infrastructure definitions on a different abstraction.
  • Volume semantics need attentionModal volumes and network file systems have their own consistency and eviction rules. Treating them like a POSIX disk leads to surprises on multi-container writes.
  • Observability is improving but partialMetrics and logs are present in the dashboard; deep tracing, structured logs and cross-function correlation still usually need an external APM or Langfuse to be complete.

Sound familiar?

Common Modal problems we fix

The Modal problems that bring people to us, and what you have once each one is fixed.

App.webapp not found

App.lookup races against deployments

A caller resolves an App by name just as a deploy is in flight, or against an environment where the app has never been deployed. The caller gets NotFoundError rather than a meaningful routing error.

Lookups resilient to deploys

GPU quota exceeded in region

GPU quota hit under load

A traffic burst scaled past the account's quota for the chosen GPU in the chosen region, and new requests queue indefinitely. The fix is sometimes a smaller GPU, sometimes an alternate region, sometimes a quota increase.

Capacity available for peak

FunctionTimeoutError exceeded 600s

Long-running jobs cut off at the limit

A fine-tune, a batch embedding run or an agent loop that ran past the configured timeout, with no checkpointing so the next retry starts from scratch.

Jobs checkpointed, timeouts realistic

ImageBuildError: pip install failed on torch

Image build breaks on dependency mismatch

A torch wheel that needs a different CUDA base, a transitive dependency that moved, or a pinned package that is no longer available on PyPI. Builds fail at deploy time, after you thought the change was going live.

Image builds reproducible

cold start 42.8s

Cold starts outside the latency budget

A large image or heavy model weights that take tens of seconds to load, with keep-warm disabled. The first user of the morning sees the full cost; everyone after that sees fine latency.

Warm-pool tuned to real traffic

VolumeNotFound in environment prod

Volume missing in the production environment

A volume that exists in staging but was never created in production, or the mount path drifted between environments. Functions crash on the first read.

Environments parity-checked

secret not set in environment staging

Secret missing after an environment rename

A secret that lives in one environment and was referenced from another, or a key renamed in the console without an alias. Functions boot and fail on the first API call.

Secrets present where they are read

DeserializationError: argument images

Large or unpickleable arguments fail

An argument that exceeds the serialisation limit, or an object that Modal cannot pickle across the container boundary. The error is at invocation time, not at build time, so it hides until production traffic hits it.

Payloads moved to volumes or signed URLs

What we do

Modal help, from cold starts to quotas

Pull us in for one slow endpoint, or let us build the whole Modal footprint for your ML workload.

Cold-start tuning
Images slimmed, model weights put on volumes rather than baked in, keep-warm pools sized to your traffic shape and the latency contract you need to hold.
GPU capacity planning
Right-sized GPU choices, region selection and quota requests set before the workload hits them, with fallbacks for when a specific GPU is temporarily tight.
Jobs and schedules
Batch pipelines with checkpointing, retries and dead-letter handling, so a long fine-tune does not start over every time the timeout trips.
App and environment hygiene
Staging and production parity, volumes and secrets mirrored, and App.lookup patterns that are safe against in-flight deploys.
Observability
Logs, metrics and traces wired to the APM you already use, with per-function cost attribution so the Modal bill matches what each product surface is actually doing.
Managed Tech
Ongoing watch on the Modal footprint, image rebuilds against new base images, GPU price and capacity tracking, and a team to call when a deploy goes sideways.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Modal alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Modal questions

Something else? Call+1 737 727 7327

Why do my Modal cold starts take so long?

Three usual drivers: a large image that has to be pulled on fresh containers, model weights baked into the image or loaded from disk instead of a shared volume, and no keep-warm pool for the function. The fix is to slim the image, move weights onto a Modal volume so they are shared across containers, and add just enough warm capacity to absorb the first requests.

How do we avoid GPU quota errors on Modal?

Watch regional capacity through the dashboard, pick GPUs sized to the actual workload rather than the largest that fits, and request quota increases before the product launches rather than after a traffic spike hits. For workloads with real peaks, a mix of GPU types with graceful fallback usually works better than one scarce SKU.

Should we use Modal or AWS Lambda for this workload?

Lambda is better when there is no GPU involved, memory stays modest and the ecosystem benefit of being on AWS matters. Modal wins when the workload needs a GPU, when iteration speed matters, or when the dependency surface is Python-heavy and Lambda's packaging limits hurt. Many teams run both.

How do we deploy a non-trivial app to Modal?

Keep images reproducible with pinned versions, store model weights on volumes so they survive deploys, use distinct environments for staging and production with the same secret names, and treat App.lookup as a routing primitive that expects to be retried. Also wire deploys through CI rather than running modal deploy from laptops.

Can Modal replace our ML inference cluster?

For many teams, yes. The hard part is usually not the GPU work itself but the moving parts around it: scheduling, retries, autoscaling and cost attribution. Modal covers those well enough to replace a modest GKE or EKS inference setup. Very high-volume real-time inference with aggressive SLOs may still want dedicated infrastructure.

How much does Modal help cost?

Our estimate is free. Walk us through the workload and the symptom, whether that is cold starts, quotas or cost, and you get a price up front before work begins.

Free estimate

Modal giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.