Running Cohere in production?

Command R+ or Rerank behaving badly? We sort it out.

Cohere's Command R+ was built for retrieval-augmented work and tool calls, and the Embed and Rerank endpoints are the quiet backbone behind a lot of enterprise RAG systems. What they share is that the API is simple until you scale it, at which point context windows get crowded, tool-call JSON drifts, and the Rerank bill climbs faster than anyone modeled. We rework the retrieval, the prompts and the request shapes until the pipeline is both correct and affordable.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

cohere-api.log

still failing

What it is

Cohere at a glance

Cohere is a Toronto-headquartered LLM provider founded in 2019 by Aidan Gomez and collaborators, with a deliberate focus on enterprise use rather than consumer chat. Its hosted models include the Command R and Command R+ generative series, Embed and Embed Multilingual for vector representations, and Rerank for scoring retrieval results.

The platform is used for retrieval-augmented generation, grounded chat, document search, classification, and agentic tool use across private deployments. Models are available through Cohere's own API, AWS Bedrock, Azure, Oracle and a self-hosted option for regulated environments. Billing is per million tokens for generation and embedding, and per search unit for Rerank.

Type
Enterprise large language model and retrieval API
First released
2021 public API, Command R+ in 2024
License
Proprietary API, open weights for some older models
Maintained by
Cohere Inc.
Typically used for
RAG, grounded chat, semantic search, document reranking, agentic tool use

The honest version

Cohere pros and cons

No technology is all upside. This is where Cohere earns its place and where it tends to cause trouble.

Where Cohere is strong

  • Built for retrieval from day oneCommand R+ ships with first-class support for grounded documents and citation spans in the response, which removes a whole layer of glue code you would otherwise write yourself.
  • Strong multilingual embeddingsEmbed Multilingual covers over 100 languages in a single vector space, which is practical for global search without maintaining per-language indexes.
  • Rerank is a real quality liftDropping a Rerank stage after a vector search usually improves top-k relevance measurably, and it is cheap enough to run on every query.
  • Deployment choice is genuineCohere, Bedrock, Azure, Oracle and private VPC options all exist and are supported in parallel, which matters for regulated customers who cannot ship data to a general-purpose endpoint.
  • Tool calling is deterministic enough to trustCommand R+ produces tool calls with clean JSON more consistently than older models, which cuts the retry-and-repair logic a lot of agent frameworks still ship with.

Where Cohere bites

  • Context window still fills up fast128k tokens sounds generous until you stuff 40 retrieved passages, chat history and a tool schema into a single turn, at which point requests start getting truncated.
  • Rerank cost climbs with trafficAt one query per user per minute it is affordable; at a search product with autocomplete-grade traffic it is the biggest line on the bill if nobody set a cap.
  • Tool-call JSON still driftsEven Command R+ produces responses that fail a strict JSON schema under long or ambiguous prompts, so your client code still needs defensive parsing and repair.
  • Smaller ecosystem than OpenAI or AnthropicSDK coverage is solid for Python and JavaScript, and thinner elsewhere. Third-party agent frameworks sometimes lag a release behind.
  • Private deployment is sales-drivenThe self-hosted path is real but gated behind enterprise contracts, so an ambitious startup cannot pilot it without going through a procurement cycle.
  • Safe mode can be aggressiveThe safety classifier sometimes refuses to answer questions that are routine for a corporate chatbot, and tuning it requires enterprise-tier access to the right parameters.

Sound familiar?

Common Cohere problems we fix

The Cohere problems that bring people to us, and what you have once each one is fixed.

ContextLengthExceeded: 131204

The prompt is bursting the context window

Too many retrieved passages, a long system prompt and multi-turn history pushed past 128k. We cut retrieval depth, summarise history, and move large-but-static material into a cached preamble.

Requests fit the window again

ToolCallParseError: malformed JSON

Tool calls return broken arguments

The model emits JSON that looks almost valid but trips a strict parser on a trailing comma or a stray string. We tighten the tool schema and add a one-shot repair pass so the client never crashes on a single bad call.

Tool calls parse cleanly

rerank latency p95 1240ms

Rerank is slowing the search endpoint

A reranker configured to score 500 candidates on every query. We cut candidate depth, batch requests, and move cheap re-use to a cache so the P95 drops below the user-facing target.

Rerank fast and cheap again

EmbedBatchTooLarge: 96 docs

Embed calls fail on large batches

A job that assumed 100 docs per request keeps hitting the 96 cap. We split the stream into correctly sized batches with retry, and keep the throughput up through concurrency rather than batch size.

Embedding pipeline runs clean

citation spans misaligned

Citations point at the wrong part of the document

Chunking or offset handling in the RAG layer drifted and citation highlights land on neighbouring passages. We fix the chunk boundaries and the offset math so UI highlights land exactly where they should.

Citations line up with sources

AzureCohere deployment quota hit

Azure-hosted Cohere is quota-capped

The deployment's token-per-minute cap is reached during the business-hours peak. We size the deployment against real traffic and move low-priority jobs off-peak.

No more mid-day throttling

safe mode filtered response

The model refuses legitimate corporate queries

Safe mode catches questions that are perfectly fine in your context. We adjust the system prompt and, where entitled, the safety settings so the model answers the real user without opening the gate to actual misuse.

Legit answers go through

What we do

Cohere help, from RAG rescue to managed operations

Engage us to fix one slow endpoint, or let us run the whole Cohere-backed pipeline.

Rescue
RAG that returns the wrong passage, Rerank that has turned the bill red, tool calls that crash clients, all triaged and stabilised first.
RAG design
Embed model choice, chunking, retrieval depth and Rerank layering tuned against your real corpus and real queries, with evaluation to prove the lift.
Tool use and agents
Tool schemas tightened, Command R+ prompts hardened, and defensive parsing added so a single malformed response does not take an agent offline.
Deployment choice
Direct API, Bedrock, Azure or private VPC picked against your data-residency and compliance constraints, with the cost tradeoffs laid out plainly.
Managed Tech
Query-cost dashboards, model-version pinning, weekly evals on a held-out set, and a US engineer to call when a new Command release lands with changed behavior.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Cohere alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Cohere questions

Something else? Call+1 737 727 7327

Why does Command R+ keep hitting the context limit?

The 128k window seems large, but RAG pipelines fill it faster than teams expect: twenty to forty retrieved passages, a long chat history, a verbose system prompt and a detailed tool schema all compound. The practical fixes are to cap retrieval depth at the number of passages the model actually uses, summarise chat history older than a few turns, and move large static material like policy documents into a cached preamble.

Should I use Cohere Rerank on every search?

On most corpora, a two-stage pipeline that retrieves 50 to 200 candidates with Embed and scores them with Rerank measurably improves top-k relevance for very little latency. The question is cost control. At autocomplete traffic rates, running Rerank unbounded is where the bill goes. A cache keyed on the query plus candidate set, plus a sensible candidate cap, keeps it affordable.

How do I stop tool calls returning broken JSON?

Tighten the tool schema with explicit types and required fields, keep tool descriptions short and precise, and lower temperature for tool-use turns. Even with Command R+ you will still see occasional malformed output, so the client should include a repair step that re-asks the model to fix its own JSON against the schema before propagating an error.

Which deployment option should we pick?

If data residency is not a blocker, Cohere's direct API is the simplest and gets the newest models first. For AWS-centric shops that need a single bill and governance, Bedrock is the obvious path. Azure and Oracle fit similar reasoning for those clouds. Self-hosted or private VPC is for regulated workloads where the data cannot leave your environment, and only makes sense once volume justifies the contract.

How much does Cohere help cost?

Quotes are free. Share the workload, the models in use and the problem you are trying to solve, and a US engineer will reply with a scoped price and a rough token and rerank-unit budget before any work starts.

Free estimate

Cohere giving you trouble?

Send over the model, the traffic shape and what has stopped working, and a US engineer will come back with a plan and a price.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.