Running vLLM?
vLLM throughput below spec or crashing under load? We ship it properly.
vLLM is the default answer when a team wants to self-host an open-weight model and keep per-token cost in their own hands. The headline throughput numbers are achievable, but only once the tensor parallel size divides the heads, the KV cache fits the batch pattern, and the request shapes do not thrash the paged attention. We tune the config against your real prompts and your real GPUs.
- Since 2015Fixing technology
- FreeEstimates
- US teamReal engineers
vllm-server.log
still failing
What it is
vLLM at a glance
vLLM is an open-source serving engine for large language models, developed at UC Berkeley and now maintained by a broad community of contributors. Its headline feature is PagedAttention, a memory manager for the attention KV cache that lets a server batch many requests at different stages of generation without fragmenting GPU memory.
It supports continuous batching, tensor and pipeline parallelism, quantisation formats including AWQ, GPTQ, FP8 and GGUF, and an OpenAI-compatible HTTP server out of the box. The project moves fast; a minor release every few weeks adds model support or performance wins, and occasionally breaks a config that worked a release ago.
- Type
- Open-source LLM inference and serving engine
- First released
- 2023 public release
- License
- Apache 2.0
- Maintained by
- vLLM community, originally UC Berkeley
- Typically used for
- Self-hosted LLM serving, high-throughput batch inference, OpenAI-compatible endpoints on your own GPUs
The honest version
vLLM pros and cons
No technology is all upside. This is where vLLM earns its place and where it tends to cause trouble.
Where vLLM is strong
Continuous batching and PagedAttentionThe scheduler fills the GPU with sequences at different generation stages, which gets throughput numbers that naive batching cannot approach.
OpenAI-compatible HTTP serverA client pointed at vLLM thinks it is talking to OpenAI, so adoption inside existing code is often a base URL change away.
Wide model and quantisation supportLlama, Mistral, Qwen, DeepSeek, Mixtral, Gemma and many more, across FP16, BF16, AWQ, GPTQ, FP8 and GGUF formats.
Tensor and pipeline parallelism built inLarge models span multiple GPUs without a separate orchestrator; the engine shards attention and feed-forward layers across the devices you give it.
Active upstreamModel support and performance fixes arrive in weeks rather than quarters, and the Discord is responsive on real bugs.
Where vLLM bites
Configuration is sharp-edgedTensor parallel size must divide attention heads, max model length must fit the model config, and KV cache memory has to be reserved up front. Wrong values do not warn, they crash.
KV cache evictions kill throughput on long contextsWorkloads with many long prompts outrun the paged attention block budget and the engine starts evicting sequences, which costs re-prefill time on the next request.
You own the GPUNo managed service here; sizing, provisioning, drivers, monitoring and availability are your problem. Hyperscalers and Together AI hide that work.
Release churn breaks configsA flag that worked at 0.6 may be renamed or defaulted differently at 0.8. Production upgrades need a rehearsal rather than a nightly bump.
Observability is minimum-viableThe built-in Prometheus metrics cover the basics; deeper tracing of per-request latency through prefill and decode needs external instrumentation.
Multi-model hosting is awkwardThe engine is one-model-per-process by design. Serving several models means several processes, each with its own GPU allocation and port.
Sound familiar?
Common vLLM problems we fix
The vLLM problems that bring people to us, and what you have once each one is fixed.
attention heads must be divisible by tensor parallel size
The server refuses to start on the chosen GPU count
A Llama 3 with 32 heads cannot shard across 3 GPUs; it has to be 1, 2, 4 or 8. The symptom is a crash on boot rather than a degraded run, which is the right place to catch it.
Parallel size matches the model
Sequence group evicted from KV cache
Throughput collapses under long-prompt load
Long contexts eat KV cache blocks, the engine evicts active sequences to make room, and the evicted ones need to prefill again on the next step. Lowering max_num_seqs or raising gpu_memory_utilization usually recovers it.
KV cache sized for the real workload
max_model_len greater than derived
A context length the config supports does not engage
The model's own config declares a shorter rope than the request asks for; raising it needs rope scaling or a different base model. Setting it blindly results in nonsense tokens.
Context length honestly supported
CUDA out of memory on launch
The engine OOMs before the first request
Reserved KV cache plus model weights plus workspace exceeds the GPU. Lowering gpu_memory_utilization, enabling a different quantisation, or adding a GPU is the lever list.
Model fits with headroom for batches
AsyncEngineDeadError
The async engine dies and never recovers
An upstream error during a request corrupts the engine state and subsequent requests all fail. A health check that restarts the server on this specific error is a load-bearing safety net.
Engine self-heals on fatal errors
quantization method 'awq' not supported
Quantised weights do not load
A checkpoint built with a different library's AWQ variant, or a quantisation format the running vLLM version does not implement yet. Pinning the version to one that matches the weights fixes it.
Quantised model loads cleanly
NCCL error: unhandled system error
Multi-GPU runs fail on inter-GPU communication
A missing NCCL version, a container without IPC namespace sharing, or GPUs that do not share NVLink in a way the shard expects. Setting NCCL_DEBUG makes the real problem visible.
Multi-GPU runs reliable
What we do
vLLM help, from one broken deploy to a tuned serving tier
One failed boot, or your whole serving stack; both get the same careful review.
01
- Deployment sizing
- GPU count, parallel size, max model length and KV cache budget tuned to your prompt distribution rather than a vendor benchmark.
02
- Throughput tuning
- Continuous batching, prefix caching, chunked prefill and quantisation set against your real workload shape.
03
- Upgrades
- Minor-version bumps rehearsed on a copy of the serving stack, with config changes reviewed before the live cut.
04
- Multi-model hosting
- Several engines behind a router with health checks, graceful restarts and per-model limits.
05
- Observability
- Prometheus, Grafana and structured logs wired up so prefill, decode and queue time are visible per request.
06
- Managed Tech
- We watch the serving tier end-to-end: GPU memory, queue depth, upstream releases and CVE notices, and act before the alert reaches your team.
How it works
From first message to fixed
1
Look Free estimate
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
2
Fix
We fix the cause, not the symptom. If something is down, getting it back up comes first.
3
Harden
Backups, monitoring, updates and access, so the same problem can't quietly come back.
4
Hand back or stay on
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
Running vLLM alongside something else?
We work on the whole stack, so the problem can be wherever it actually is.
Why is our vLLM throughput lower than the published benchmark?
Benchmarks assume short prompts and bounded max tokens on a specific GPU. Real workloads with long contexts, mixed prompt lengths and large max_tokens behave differently. The three usual levers are max_num_seqs sized to the KV cache budget, chunked prefill off or on depending on latency sensitivity, and prefix caching enabled when prompts share a system message. Measuring on your prompts, not toy ones, is the only reliable path.
How do we choose a tensor parallel size?
It has to divide the number of attention heads and ideally match the number of GPUs available. For a 32-head model, valid sizes are 1, 2, 4, 8. Larger is not always better; inter-GPU communication starts to dominate for small batch sizes, so the right answer is usually the smallest size that fits the model with room for the KV cache.
Should we use AWQ or FP8 quantisation?
AWQ is more mature and gives roughly 4-bit weights with good quality retention on most models; FP8 is faster on Hopper and Ada GPUs that support it in hardware and gives near-FP16 quality with half the memory. For older GPUs AWQ wins; for H100 or L40S FP8 is usually the better call, workload permitting.
How do we upgrade vLLM safely in production?
Pin the version, rehearse the upgrade on a copy with the same model and config, check the release notes for renamed flags, and run an eval set through both versions before cutting over. The project moves fast and config compatibility is best-effort rather than guaranteed.
How much does vLLM help cost?
Estimates land free of charge. Tell us the model, the GPUs, the real traffic and where it hurts, and the scope and price come back promptly.