Running LiteLLM?

LiteLLM proxy or budgets misbehaving? We unstick it.

LiteLLM lets every provider pretend to be OpenAI, and the proxy variant is where most teams actually run it: a single gateway with keys, budgets and model lists in front of Anthropic, OpenAI, Bedrock, Vertex and a dozen others. When the config stops matching the running state, spend budgets fail to enforce, or the database behind the proxy chokes on virtual key traffic, we go through it and put the gateway back under control.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

litellm-proxy.log

still broken

What it is

LiteLLM at a glance

LiteLLM is an open-source library and proxy server that puts a unified OpenAI-format interface in front of more than a hundred LLM providers. You call chat.completions.create against LiteLLM, it figures out whether the request should go to Anthropic's API, Bedrock, Vertex AI, OpenAI, Azure OpenAI or an open-weights endpoint, and returns the response in OpenAI shape regardless.

The library form embeds directly in Python or Node applications; the proxy form runs as a service with its own Postgres database, virtual keys, team-level budgets, routing policies and guardrails. It is BSL-licensed with an enterprise tier for features like SSO, audit logs and advanced guardrails. Most teams end up on the proxy because budgets, keys and spend logging are not something you want duplicated across every application.

Type
LLM proxy and SDK, OpenAI-format gateway over many providers
First released
2023
License
Business Source License (BSL) 1.1, enterprise tier available
Maintained by
BerriAI
Typically used for
Multi-provider gateway, virtual keys, spend tracking, model routing, guardrails

The honest version

LiteLLM pros and cons

No technology is all upside. This is where LiteLLM earns its place and where it tends to cause trouble.

Where LiteLLM is strong

  • OpenAI format everywhereCode that was written against the OpenAI SDK works unchanged against Claude, Gemini, Bedrock or Mistral through LiteLLM, which is the single biggest reason teams adopt it.
  • Virtual keys and team budgetsThe proxy issues scoped API keys with per-team, per-user and per-model spend limits, which solves a real governance problem at larger companies.
  • Routing with fallbacksModel groups, latency-based routing, context-size fallbacks and cooldowns on failed deployments cover most of what a production gateway needs.
  • Self-hostableThe proxy runs in your own cluster with your own Postgres, so prompts and completions stay on infrastructure you control.
  • Guardrails and PII maskingPre-call and post-call hooks for redaction, moderation and audit are built in rather than something you glue on with middleware.
  • OpenTelemetry out of the boxSpend logs and traces can go to Langfuse, LangSmith, Datadog or plain OTLP without a custom exporter.

Where LiteLLM bites

  • Proxy adds a hopEvery request pays the network latency of the gateway, which is usually fine and occasionally painful on streaming workloads with tight tail-latency budgets.
  • Postgres is a real dependencyThe proxy writes every request to Postgres for spend logging and key tracking, so connection pool tuning and vacuum scheduling become part of your LLM platform.
  • Config drift happens fastThe config.yaml is the source of truth, but admins also edit teams and keys through the UI, so the two can drift unless you treat the UI as read-only.
  • BSL is not quite open sourceBusiness Source License restricts commercial hosting of competing services, which is fine for internal use and trips up some procurement teams.
  • Provider feature parity is partialPrompt caching, tool use and structured output are implemented differently per provider, and LiteLLM papers over the differences unevenly.
  • Upgrade cadence is fastReleases land several times a week with real breaking changes, so pinning versions and testing the proxy config on upgrade is not optional.

Sound familiar?

Common LiteLLM problems we fix

The LiteLLM problems that bring people to us, and what you have once each one is fixed.

model_group has no healthy deployments

Every deployment in a group is on cooldown

A model group of two or three backends all hit upstream errors around the same time, every one entered cooldown, and the router now returns errors for the whole group until a cooldown expires.

Cooldowns tuned, deployments healthy

BudgetExceededError: team exceeded monthly cap

Team budget blown mid-month

A team spent through its allocation faster than planned, and the alert fired after the cap rather than before. Either the budget is wrong or the application needs cheaper routes for the bulk of traffic.

Budgets aligned with real usage

too many connections to postgres

Spend log database is saturated

A traffic spike multiplied the database writes beyond the connection pool, and the proxy starts failing requests not because the model is down but because it cannot log the result.

Pool sized for the real traffic

config reload failed: unknown field

Config schema changed on upgrade

A field that was accepted on the previous version is rejected, or a deprecated field was removed in a minor release, so the config.yaml reload fails and the proxy keeps running the old state silently.

Config schema matched to the running version

AccessDeniedException on Bedrock Claude

Bedrock IAM policy is wrong

The proxy's IAM role lacks InvokeModel on the specific Anthropic ARN, or the model is not enabled in the region the proxy routes to. Most teams discover this in production because test traffic landed elsewhere.

IAM aligned with the real routes

Invalid proxy_server_token

Virtual key not accepted

A rotated master token that was not pushed to every consumer, or a team key with no models allowed. The error is the same at the proxy edge regardless of root cause.

Keys rotated end to end

pii_masker blocked request, no response returned

Guardrail silently swallows the call

A PII or moderation guardrail that matched a false positive and returned nothing, with no clear error to the caller. The application sees a hang rather than a redaction event.

Guardrails surface clear errors

context_length_exceeded (220k tokens)

Context-size fallback not firing

A request longer than the primary model's window that should have fallen through to a larger-context model, but the fallback rule did not trigger because the group was configured on cost rather than context size.

Context fallbacks configured

What we do

LiteLLM help, from proxy config to budgets

Call us for a single failing route, or let us run the whole LiteLLM gateway.

Proxy setup
Config.yaml, Postgres, Redis and routing worked out so the gateway is a thing you deploy with the rest of the platform rather than a one-off.
Virtual keys and budgets
Team-level keys with real spend limits, rate limits and model allowlists, plus alerting that fires before the cap not after.
Model routing
Model groups, context-size fallbacks, latency routing and cooldown tuning set from real traffic rather than defaults.
Bedrock and Vertex integration
AWS and GCP identity wired through the proxy correctly, with IAM scoped to the specific models in use and audit trails you can read.
Observability
Langfuse, LangSmith, Datadog or plain OTLP exporters set up, so spend and latency live in the tool your team already uses.
Managed Tech
Day-to-day care of the proxy: upgrades without surprises, config changes reviewed before they land, database kept healthy, and someone to call when a provider decides to rotate an endpoint overnight.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running LiteLLM alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

LiteLLM questions

Something else? Call+1 737 727 7327

Should we use LiteLLM as a library or as a proxy?

Use the library when there is exactly one application, one owner and no spend governance. Use the proxy when there are multiple teams or services, budgets to enforce, virtual keys to issue, or audit requirements. Nearly every company past the first few services ends up on the proxy because duplicating key management is worse than running Postgres.

Why do our LiteLLM budgets not stop spend?

Two common reasons: budgets are set per-key but spend is going through a different key, or async spend logging is backed up so the budget check does not see recent usage yet. Confirm which key each caller is using, then check the Redis or Postgres backlog on the spend log tables.

How does LiteLLM compare with OpenRouter?

OpenRouter is a hosted gateway you call over the internet; LiteLLM is software you run yourself. OpenRouter wins on zero-ops and model-availability speed. LiteLLM wins when prompts cannot leave your infrastructure, budgets need to enforce per team, or you want to run the gateway in the same VPC as the application.

Can we run LiteLLM in front of AWS Bedrock and Azure OpenAI?

Yes, and it is a common pattern. Both providers plug in as model backends with their own auth, and the proxy exposes them under OpenAI-format endpoints. The usual trap is IAM or managed identity scoped to too few models, so new deployments fail silently until the policy is updated.

How do we upgrade LiteLLM without breaking production?

Pin the exact version, keep the config.yaml under source control, and run the target release in a staging proxy with your real config before cutting over. The schema does move between minor versions, and so does the behaviour of fallbacks and cooldowns, so rehearsing the upgrade is cheaper than rolling it back at midnight.

How much does LiteLLM help cost?

Quotes are free. Describe the state of the proxy and what is going wrong, and a price comes back before we touch the config.

Free estimate

LiteLLM giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.