Running LangSmith?

LangSmith traces dropping or quota hit? We sort it out.

LangSmith is everywhere in production LLM code built on top of LangChain, and the failure modes are specific: run trees that never close, projects that quietly hit a monthly quota, feedback scores that will not attach to the right trace, and silent sampling that loses the one call you wanted to look at. We fix the plumbing and tune the client so the traces you need are actually there.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

langsmith-client.log

still broken

What it is

LangSmith at a glance

LangSmith is LangChain's hosted observability and evaluation platform for LLM applications. It captures nested run trees from LangChain, LangGraph and plain LLM calls, stores datasets of inputs and outputs, and runs evaluators (deterministic, LLM-as-judge or human) against them.

It is the default observability layer most LangChain users reach for because the integration is invisible when things work. Traces from LangGraph agents, tool calls, retrievers and streaming chat models all land in the same project with the same shape. Pricing is per-trace with a generous free tier, and there is a self-hosted enterprise option for teams that cannot send prompts off their infrastructure.

Type
LLM observability, tracing and evaluation platform
First released
2023
License
Proprietary SaaS, self-host enterprise tier available
Maintained by
LangChain, Inc.
Typically used for
Agent tracing, dataset-driven evals, prompt debugging, regression testing

The honest version

LangSmith pros and cons

No technology is all upside. This is where LangSmith earns its place and where it tends to cause trouble.

Where LangSmith is strong

  • Zero-config for LangChain and LangGraphSet two environment variables and nested runs appear in the UI with the right parent and child relationships, including tool calls and streaming tokens.
  • Honest run tree modelThe data model reflects how agents actually execute: a parent run with child LLM calls, retriever calls, tool calls and sub-agents, each with its own inputs, outputs and tokens.
  • Datasets and evaluators are first classTurning a captured run into a dataset example takes a click, and evaluators run against that dataset the same way regardless of model.
  • Works with plain OpenAI and Anthropic SDKsThe wrap_openai and wrap_anthropic helpers give you traces without adopting LangChain, which matters for teams that only use parts of the ecosystem.
  • Human feedback loopThumbs up or down, free-text annotations and reviewer queues attach directly to trace IDs, so the signal flows into datasets without a separate pipeline.
  • Hub for promptsPrompts live in a versioned registry rather than in source files, which makes rolling back a prompt change a configuration step rather than a deploy.

Where LangSmith bites

  • Per-trace pricing adds up fastHigh-volume agents with deep tool-use trees burn through quota quickly because each tool call is a run. The free tier runs out, the plan jumps, and nobody warned finance.
  • Sampling is a blunt instrumentSetting a sample rate drops whole traces at random, which is cheap but means the one failing call you wanted to debug may not be captured.
  • Run tree context propagation is fiddlySpawning background tasks or threads from an active run often loses the parent run ID, and orphaned runs land in the wrong project without complaint.
  • Self-host is enterprise-only and heavier than it looksThe self-hosted stack needs Postgres, Redis, ClickHouse and object storage, with a license that is not aimed at hobbyists.
  • Eval regressions are only as good as the datasetA stale dataset or a changed tool schema makes evaluator scores drift even when the model has not moved, which erodes trust quickly.
  • Vendor coupling to LangChainThe ergonomics are best when you are already on LangChain and LangGraph; outside that stack the integration is manual and more work to maintain.

Sound familiar?

Common LangSmith problems we fix

The LangSmith problems that bring people to us, and what you have once each one is fixed.

429 monthly trace quota exceeded

Project hit its monthly trace limit

A burst of agent traffic or a chatty tool loop quietly burned through the plan allowance, so traces from the rest of the month land nowhere and alerts go quiet.

Quota tuned, sampling by trace type

run tree ended before child run completed

Child runs are being orphaned

Asynchronous tool calls that outlive the parent run, or a context manager that closes too early. The orphaned runs show up in the wrong place or not at all, and token totals look low.

Context carried across async boundaries

parent_run_id does not match active run tree

Context propagation is broken across threads

Background jobs or worker threads inherit no tracing context, so nested calls either start a new root run or fail to post entirely. Usually a missing contextvar propagation step.

Threads carry the trace context

feedback.create: run not found

Feedback fails because of eventual consistency

Scores are written seconds after the run completes but before the trace has finalised server-side, and the API rejects the write. Retries with backoff usually fix it; blind writes do not.

Feedback attaches reliably

project not found, traces routed to default

Traces are landing in the wrong project

An environment that was never updated after a project rename, or a worker running an older env file. Everything looks fine until someone opens the production project and sees nothing recent.

Routing fixed across all services

OUTPUT_PARSING_FAILURE on 14 of 120 examples

Evaluators are failing on parseable output

An evaluator that expected strict JSON while the model returned a code fence, or a schema that drifted after an LLM upgrade. The failures count against scores rather than being surfaced as evaluator bugs.

Evaluators robust to real outputs

replay: tool schema changed since trace captured

Replaying old traces against new tools breaks

A tool whose parameters changed names or types since the dataset was built, so the replay rejects the input before the model is even called.

Dataset examples migrated

wrap_openai TypeError: unexpected keyword

The OpenAI wrapper is out of sync with the SDK

A new argument in the OpenAI Python SDK that the LangSmith wrapper has not yet accepted, so plain chat completion calls raise as soon as you add the wrapper.

Client versions aligned

What we do

LangSmith help, from rescue to eval pipelines

Bring us one leaking trace, or let us set up the whole LangSmith footprint.

Trace plumbing
Context propagation across async code, threads and background workers, with datasets tagged so the right traces are findable.
Quota and sampling
Sampling by project, trace type and user, so critical production flows are kept while chatty background work is thinned.
Evaluation design
Evaluators that survive schema drift and model upgrades, with CI jobs that fail a pull request when a regression is real.
Prompt hub integration
Prompts pulled from the hub at runtime with proper fallbacks, and releases coordinated with trace tags so rollbacks are one click.
Self-host setup
Enterprise self-hosted deployments on your cloud, with Postgres, Redis, ClickHouse and object storage sized to the workload.
Managed Tech
Ongoing watch on trace volume, evaluator health, quota burn and client library upgrades, so the observability layer itself does not become the incident.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running LangSmith alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

LangSmith questions

Something else? Call+1 737 727 7327

Why are my LangSmith traces missing or incomplete?

The most common patterns are context lost across async boundaries, background threads that never inherit the parent run, and aggressive sampling that drops exactly the trace you wanted. Check the project matches the one the env var points to, then look for orphaned runs in the project list, then confirm the client library is current.

How do we keep LangSmith costs under control at scale?

Sample by trace type instead of blanket sampling, so chatty low-value loops are thinned while user-facing flows stay fully captured. Tag traces with feature names so you can see which product surface is burning quota. Move heavy evaluation runs onto a separate project with its own budget.

Can we use LangSmith without LangChain?

Yes. The wrap_openai and wrap_anthropic helpers give you nested run trees against the plain SDKs, and the @traceable decorator works around any Python function. You lose some automatic instrumentation but the data model and UI are identical.

Should we self-host LangSmith?

Only if there is a real reason prompt and completion data cannot leave your infrastructure. The self-hosted stack is enterprise-tier, heavy to operate, and priced accordingly. Most teams are better served by the cloud offering with sensible tagging and access controls.

How do we run regression tests against LangSmith datasets?

Promote production traces into a labelled dataset, define evaluators that check the output shape and quality, and wire them into CI so a pull request that moves a prompt or model is scored before it merges. The key is keeping evaluators stable even when outputs change in benign ways.

How much does LangSmith help cost?

We quote for free. Tell us the symptom, whether it is a missing trace, a quota alert or an evaluator that will not stop failing, and a price comes back before any billing starts.

Free estimate

LangSmith giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.