LangSmith Engine Tutorial: Autonomous Agent Debugging in Production

Introduction

If you run an AI agent in production, you already know the pain: the agent generates thousands of traces a day, a few of them fail in strange ways, and finding the pattern means hours of scrolling through nested spans. This tutorial shows you how to set up LangSmith Engine, LangChain’s in-platform agent that watches your production traces for you, groups failures into named issues, diagnoses the root cause against your codebase, and drafts fixes and evaluators for your review. It is written for engineers who already ship an agent with LangSmith tracing, and who want the debugging loop to run itself.

Quick answer

LangSmith Engine is an autonomous agent that monitors your production LangSmith traces, clusters failures into named issues, diagnoses root causes against your code, and drafts fixes plus eval coverage for review.

Why Engine exists: the evidence

The agent development lifecycle has always been manual at scale. LangChain’s own framing of the problem is blunt: individual trace review does not reveal patterns, error recurrence is invisible across thousands of traces, and building ground-truth eval examples from production data is tedious enough that teams skip it (Introducing LangSmith Engine). That is the exact friction Engine was built to remove.

The numbers since launch are striking. LangSmith Engine launched in public beta in May 2026, and by August 2026 it had scanned more than 60 million production traces and found over 20,000 issues across customers’ agents, saving what LangChain describes as tens of thousands of hours of engineering work (New in LangSmith Engine). Engine v2, announced at LangChain Interrupt 2026 in New York in September 2026, improved on that baseline: it scores over twice as well on LangChain’s internal IssueBench benchmark for issue detection, and its proposed fixes score 25 percent better on the public Terminal-Bench benchmark.

Early production users are named, too: teams at Cogent, Harmonic, and Campfire have used Engine to resolve issues affecting thousands of traces. Nancy Jin, AI Engineer at Campfire, said of it: “Engine brings observability and triage into one place. I was amazed by the quality of its root-cause diagnostics and its drafted proposed fixes.” The pattern these teams report is consistent: less time hunting through traces, a shorter path from broken behavior to a deployed fix.

Engine watches five signal types in your traces: explicit errors such as tool call failures and timeouts, online evaluator failures, trace anomalies like latency spikes, token blowouts and unexpected step counts, negative user feedback, and unusual behaviors such as users asking questions the agent was never built to answer. When it spots a pattern across multiple traces, it clusters them into a single named issue instead of surfacing each failure on its own.

Prerequisites

• A LangSmith account on the Plus or Enterprise plan (Engine is not available on the free tier).
• An agent already sending production traces to LangSmith, via the official SDK (pip install -U langsmith) or any framework with LangSmith tracing enabled.
• Python 3.9 or newer for the setup snippets below.
• Optional but recommended: GitHub repository access, so Engine can draft pull requests.
• Familiarity with the trace, dataset, and experiment concepts in LangSmith. If you are building the agent itself from scratch, start with our hands-on guide to building a LangChain RAG workflow in n8n, which covers tracing-enabled agent design.

Step 1: Get production traces flowing into LangSmith

Engine has nothing to analyze without trace volume, so the first real step is making sure your production agent emits traces to the project you will point Engine at. Install the SDK and point it at LangSmith with the standard environment variables documented in the official SDK README:

pip install -U langsmith

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=lsv2_your_key_here
export LANGSMITH_PROJECT=my-support-agent-prod

With LANGSMITH_TRACING=true, any run through LangSmith’s traced clients is logged automatically. If you want an explicit trace boundary around your agent’s top-level entry point, wrap it with the SDK’s @traceable decorator:

from langsmith import traceable

@traceable(name="support-agent-run")
def run_support_agent(user_message: str) -> str:
    # your agent logic here: LLM calls, tool calls, retrieval
    return final_answer

Two details matter for Engine specifically. First, negative user feedback is one of Engine’s five signal types, so log feedback on runs where your product already captures thumbs up or down. Second, keep the project dedicated to one agent in production; Engine clusters issues per project, and mixing staging and production traffic in one project dilutes the signal.

Step 2: Enable Engine on your project

With traces flowing, connect Engine. LangChain’s documented path is simple: connect a tracing project, optionally connect your repository, and Engine begins surfacing issues from your production traces automatically. On a Plus plan, you enable Engine with a few clicks inside the LangSmith app at smith.langchain.com; Enterprise teams enable it through their account team (New in LangSmith Engine).

Connect the repository if you can. Engine’s deepest value, the drafted pull request, requires repository access: without it, Engine still diagnoses issues and proposes evaluators, but it cannot open the fix for you. For self-hosted deployments, Engine’s orchestration runs inside your own VPC and makes requests to LangSmith Intelligence, LangChain’s managed zero-data-retention inference service, which keeps enterprise security and compliance teams satisfied.

Step 3: Read an Engine issue end to end

To know what “done” looks like, walk through the canonical example LangChain documents in the Engine launch post. Imagine your agent is a customer support bot. Engine detects a cluster of traces where users ask about canceling their subscription: the agent responds, but your online evaluators score the responses as failures and user feedback is negative. Latency is normal, so no systems alert fired.

Engine surfaces this as a single named issue, “Agent fails to handle subscription cancellation requests accurately,” with a severity (high, affecting 12 percent of support sessions that week), a timeline (started four days ago, correlating with a recent deployment), and links to the specific traces as evidence. With your repository connected, Engine reads the relevant code and identifies the root cause: the cancellation tool description is ambiguous, so the agent attempts cancellation when users are only asking about their options. Engine then drafts a pull request with a targeted fix to the tool description.

This is the documented workflow, and your own issues will follow the same shape: named issue, severity and blast radius, incident timeline with recurrences, linked evidence traces, root cause, and a proposed action. When you open your first Engine issue, check these four artifacts in order before doing anything else: the diagnosis with its evidence, the proposed fix, the ground truth examples Engine formatted from your failing runs, and the ongoing monitoring Engine attached to catch regressions.

Step 4: Accept the three resolution actions

For every issue it surfaces, Engine proposes three resolution actions, and understanding all three is what turns a one-off fix into durable reliability.

First, open a pull request. With repository access, Engine drafts a targeted code or prompt change and opens it against your repo. You review and merge; nothing ships without your approval.

Second, create a custom online evaluator. Engine proposes an evaluator scoped to the exact problem. If the failure pattern recurs after the fix ships, the issue is resurfaced automatically with updated details. This is the piece most teams skip when debugging manually, and it is the piece that prevents the same bug from silently returning.

Third, add to your offline eval suite. Engine pulls the failing production traces into a dataset of ground truth examples with per-example criteria defining what the correct output should contain, ready to run in your offline eval workflow. Your production failures become the regression tests that keep them out.

Step 5: Wire Engine into your team’s workflow

Engine also ships workflow integrations that move it out of the LangSmith UI and into the tools your team already uses.

Slack notifications. Engine can notify agent engineers about detected issues in Slack, so investigation starts minutes after detection instead of at the next scheduled trace review.

Linear issues. Engine can automatically open Linear issues for detected problems, which puts agent reliability work in the same backlog and prioritization flow as the rest of engineering.

Reduced Analysis mode. Cost-sensitive teams can scan fewer traces with Engine, trading some detection coverage for lower spend while keeping the loop running.

Issue Hygiene. Engine automatically closes stale issues that have stopped appearing in your traces, so engineers spend triage time on active, impactful problems rather than a growing list of ghosts.

Engine v2 red teaming and fix validation. If your agent runs on LangSmith Deployment, Engine v2 goes further: it proactively red-teams your agent by generating hypotheses about issues that have not appeared in production yet, tests those hypotheses against your agent, and surfaces confirmed failures for review. It also detects harder-to-spot issue types, including performance trends across error rate, latency, and cost, and inefficient agent behavior like repetitive tool calls or unnecessarily long trajectories. For proposed fixes, Engine v2 reproduces the failure, runs the offending inputs against your agent to confirm the issue, tests candidate fixes against a broader eval set until one resolves it, and presents the validated change for review, where you can open the pull request with a click.

Troubleshooting

Engine is not surfacing any issues on my project

This is almost always a signal problem, not an Engine problem. Engine needs trace volume plus at least some of its five signal types. If your project only has clean traces with no evaluator feedback and no user feedback, there is nothing for the clustering to latch onto. Fix it by confirming real production traffic is landing in the project Engine is watching, attaching online evaluators to your key flows, and logging user feedback on runs. Also confirm you are on a Plus or Enterprise plan; Engine does not run on the free tier.

Engine diagnoses issues but never drafts pull requests

Pull request drafting requires repository access. If you enabled Engine with only the tracing project connected, you get clustering, diagnosis, evaluators, and datasets, but no PR. Revisit the Engine settings for your project and connect the repository that holds your agent’s code and prompts. After connecting, Engine needs a few fresh traces before it can read code in context, so give it a production cycle before judging.

Red teaming and automated fix validation are missing

These are Engine v2 capabilities tied to LangSmith Deployment, and the red teaming plus automatic fix validation features rolled out as a private beta for teams already on LangSmith Deployment. If your agent runs elsewhere, you still get issue detection, root cause diagnosis, PR drafts, evaluators, and datasets, but not the proactive hypothesis testing or the pre-review fix validation. Migrating the agent to LangSmith Deployment is the documented path to unlock them.

Self-hosted Engine cannot reach LangSmith Intelligence

On self-hosted deployments, Engine’s orchestration runs inside your VPC and calls out to LangSmith Intelligence for inference. If issues stall at the analysis stage, check that your VPC allows egress to the LangSmith Intelligence endpoints and that your service credentials are current. This is the one Engine failure mode that looks like an Engine bug but is a network policy; the LangSmith status page and your VPC flow logs will confirm it faster than re-enabling Engine.

Issue list is noisy with low-value findings

Turn on Reduced Analysis mode if cost is the concern, but for noise the better lever is Issue Hygiene plus evaluator tuning. Engine auto-closes stale issues that stop appearing in traces, and low-signal issues usually trace back to an overly strict online evaluator. Tighten the evaluator’s criteria to the behavior you actually care about, and the issue list converges on problems worth fixing. The v2 benchmark gains, over 2x better issue detection on IssueBench, specifically target this higher-signal surfacing.

FAQ

See the FAQ in this article’s front matter for the five most-asked questions: what Engine changes in your workflow, which plan you need, whether Engine can ship fixes by itself, whether the repository connection is required, and how Engine v2 differs from the original.

Conclusion

LangSmith Engine turns the most tedious part of running an AI agent, reading production traces to find patterns, into an autonomous loop that ends at your review queue: named issues with evidence, drafted fixes, new evaluators, and growing regression datasets. Connect a tracing project, link your repository, wire up Slack or Linear, and let the September 2026 v2 capabilities, red teaming and automated fix validation, harden the loop further.

As a next step, if your agent talks to external tools, read our guide on how MCP servers work to understand the protocol layer your agent’s tool calls run on, and our self-hosted n8n setup on Ubuntu if you want a private automation layer in front of your LangSmith-traced agents.

Frequently asked questions