Building RAG on Chroma?

Chroma persistence or collections acting up? We bring it back.

A prototype that ran out of PersistentClient on a laptop has become a production retrieval layer, the server-mode container keeps reporting a client API version it does not speak, and a collection that held forty thousand vectors last week now returns zero matches without any error in the log. We walk the storage layout, check the SDK and server versions against each other, and turn the retrieval back into something a chatbot can rely on.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

chroma-server.log

still broken

What it is

Chroma at a glance

Chroma is an open-source embedding database written in Python with a Rust core for the index. It is designed to be the smallest sensible thing between an application and a vector store, so RAG prototypes can start with an in-memory client and move to a persistent store or a server deployment without changing the application code. Collections hold embeddings, documents, IDs and metadata together, and queries combine vector similarity with structured filters.

The project was created by Jeff Huber and Anton Troynikov in 2022 and has since grown a managed cloud offering alongside the open-source code. Chroma's defaults are forgiving, which is part of the appeal and part of the trouble, because teams often ship with the ephemeral in-memory client, with default embedding functions that do not match the ingest pipeline, or with persist_directory pointing at a container path that is wiped on redeploy. Client and server versions moved rapidly between 0.4, 0.5 and 0.6, so a mismatched SDK is one of the single most common failure modes in the wild.

Type
Open-source embedding database for AI applications
First released
2022
License
Apache 2.0 (open-source core), paid managed cloud
Maintained by
Chroma Inc. and open-source contributors
Typically used for
RAG for chatbots and agents, semantic search over documentation, prototype vector stores that graduate into production

The honest version

Chroma pros and cons

No technology is all upside. This is where Chroma earns its place and where it tends to cause trouble.

Where Chroma is strong

  • The friendliest first ten minutes in vector searchchromadb.Client(), add(), query(). That is the whole surface area, and it is enough to build a working retrieval demo in an afternoon without reading a cluster management guide.
  • One API across in-memory, persistent and server modesThe same Python calls work against an ephemeral client, a PersistentClient on disk, or a running Chroma server, so the move from laptop to production does not force a rewrite.
  • Metadata and full-text filters alongside vector searchwhere and where_document filters narrow a query before the nearest-neighbor step, which keeps multi-tenant and permissioned retrieval honest without a second data store.
  • Open-source under Apache 2.0Running Chroma inside your own infrastructure is a first-class path, and the managed cloud is an option rather than a requirement, which keeps procurement happy.
  • Strong integration with LangChain and LlamaIndexBoth frameworks treat Chroma as a reference vector store, so the retriever wiring is maintained and the examples in the docs usually work out of the box.
  • Honest, readable codebaseThe project is small enough that you can read the HNSW index wrapper and the persistence layer when a problem gets interesting, which is a comfort when something behaves unexpectedly.

Where Chroma bites

  • Client and server versions are not always in lockstepA pip install chromadb that pulls 0.5.x against a running 0.4.x server will greet you with cryptic errors. The SDK pins and the server image need to agree, and they do not negotiate.
  • SQLite under the hood has real limitsChroma persists metadata in SQLite, which handles small and medium workloads calmly and starts to lock under heavy concurrent writes. Production write paths need to understand that.
  • Defaults are too convenientNo persist_directory means the collection lives in /tmp and dies with the container. The default embedding function is a small onnx MiniLM model that quietly disagrees with whatever the ingest pipeline is using.
  • Scaling horizontally is limitedChroma was designed for single-node simplicity, and sharding across nodes is still rough compared to Weaviate or Qdrant. Large corpora push you toward the cloud offering or a different engine.
  • Tenants and multi-tenancy arrived lateThe default_tenant and default_database model is recent, and older deployments drift when the server is upgraded past the version that introduced them.
  • Observability is minimalMetrics and slow-query logging are thinner than a database carrying production retrieval warrants, and most teams end up instrumenting the client side themselves.

Sound familiar?

Common Chroma problems we fix

The Chroma problems that bring people to us, and what you have once each one is fixed.

client API 0.5.0 contacting server 0.4.24

Client and server versions do not match

The Python SDK was upgraded in CI without pinning, the server image is still on an older tag, and every call either fails outright or quietly ignores a new parameter.

Client and server pinned together

no persist_directory set, data in /tmp

Collections vanish after a redeploy

A Chroma server started inside a container with the default ephemeral storage. Every restart wipes the collection and the ingest job has to run from scratch while users see empty answers.

Persistence on durable storage

sqlite3.OperationalError: database is locked

Writes are contending on SQLite

Multiple ingest workers hammering upsert in parallel, long-running queries holding read locks, or a backup script reading the file at the wrong moment. SQLite does not fan out.

Writes serialized or moved off SQLite

Embedding function returned dimension 1536

Ingest and query embeddings are from different models

The collection was built with an onnx MiniLM default, the query path has been upgraded to text-embedding-3-small, and the two coordinate spaces have nothing to do with each other.

One embedding function across read and write

NoIndexException: Index not found on 'docs'

The HNSW index disappeared

A collection whose index file was lost during a volume move, or a restore that pulled the SQLite file without the parquet shards next to it. Queries error until the index is rebuilt.

Index rebuilt, storage layout understood

ConnectionError: tenant 'default_tenant'

Tenant model changed under a running server

A server upgrade introduced tenants and databases, and the client is still calling the old flat API. Nothing is actually broken, the namespacing just moved.

Client talks to the correct tenant

ef_construction cannot be smaller than M

HNSW parameters were set incoherently

A collection created with construction parameters copied from a different engine, where the numbers are valid but mean something else, so the index will not build.

HNSW parameters measured on your corpus

500 Internal Server Error on upsert batch

Large upserts bring the server down

A batch that is too big for the server's memory, or a payload with document text that pushes past the request limit. The server returns 500 and the worker retries the same batch.

Batching sized to the server

What we do

Chroma help, from prototype to steady production

Pick up the phone for one broken collection, or let us run retrieval end to end.

Prototype to production
Moving from an in-memory Chroma on a notebook to a persistent or server deployment that survives redeploys, backups and upgrades.
Embedding alignment
One embedding function across ingest and query, versioned, reproducible, and measured against a labeled relevance set.
Persistence and backup
persist_directory on durable storage, backups of the parquet shards and SQLite file together, and restores that have actually been tested.
Client and server version fixes
SDK pinned to the right server image, tenant and database settings matched, and upgrades planned rather than discovered.
RAG quality review
Chunking, embedding and query reviewed end to end, with recall and precision measurements before and after each change.
Managed Tech
Dashboards on collection size, query latency and persistence health, with a human to call when a redeploy wipes the data at a bad hour.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Chroma alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Chroma questions

Something else? Call+1 737 727 7327

Why do our Chroma collections disappear after a redeploy?

Almost always because persist_directory was never set, or it points at an ephemeral container path. The in-memory client is the default, and a path under /tmp is wiped when the container restarts. Point persistence at a mounted volume, move the parquet and SQLite files together whenever the storage is restored, and test that a fresh container can read the existing collection before you trust it.

Can we use Chroma in production, or should we move to something else?

Chroma runs production RAG for a lot of teams, especially at the small and medium end. The honest limits are concurrent write throughput, horizontal scaling and operational tooling, where Weaviate, Qdrant or Pinecone have had more time to mature. If a single-node server with SQLite under the hood matches your write pattern, it is a perfectly reasonable choice.

Why are our Chroma results suddenly worse?

In almost every case the embedding path changed. A library upgrade swapped the default embedding function, the chunker started splitting documents at a different boundary, or a retrieval parameter in LangChain drifted. We rebuild a labeled evaluation set, measure recall before touching the system, and isolate the step that regressed.

How do we upgrade Chroma without losing collections?

You pin the client SDK to match the target server, take a cold snapshot of the persist_directory, run the new server pointed at a copy of that data first, confirm the collections load and the queries return sensible results, and only then swap the production traffic. The tenants and databases model has changed enough between minor versions that this is worth doing even when nothing looks broken.

Can you take over a Chroma deployment the last team set up?

Yes. We inventory the collections and tenants, check whether persistence is actually durable, align the embedding function across read and write paths, rotate any API keys used by the embedding provider, and bring the deployment under version control so the next engineer does not inherit a mystery.

How much does Chroma help cost?

The estimate is free. Describe the deployment, the collection size and what retrieval is doing wrong, and you get a fixed quote before anything touches production.

Free estimate

Chroma giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.