LLM Evaluation for AI Agents: The Production Stack for 2026 (LangSmith vs Langfuse vs Braintrust)

Your agent worked perfectly in the demo. It has been silently degrading for three weeks. The customer found it before you did. This is the LLM evaluation stack that catches the regression before the screenshot lands in your inbox - the same one I implement with paying clients running production AI agents on LangGraph, CrewAI, and custom orchestrators.

Why Agent Evals Are Different from LLM Evals

An LLM eval asks: did the model give the right answer to this prompt? An agent eval has to ask three more questions. Did it call the right tools? Did it call them in the right order? Did it recover when a tool failed? An LLM produces a string; an agent produces a trajectory - a sequence of decisions, tool calls, and intermediate states. A correct final answer can hide a broken trajectory that costs ten times more or fails on the next prompt.

This is why teams that bolt LLM-style evals onto agents miss regressions. Their eval reports green while their token bill silently doubles because the agent now retries every tool call twice. The fix is to evaluate the path, not just the destination.

LLM Evaluation: The Four Layers Every Production Agent Needs

Every agent I have shipped to production in the last 18 months runs all four of these. Skip a layer and you create a blind spot.

Layer 1 - Unit evals. Test individual components: a single tool call, a single classifier, a single extraction prompt. Fast (under 5 seconds), runs on every commit, catches prompt-template regressions.

Layer 2 - Trajectory evals. Test the full agent on a fixed dataset of 50-200 real prompts. Compare the trajectory (tools called, order, intermediate state) against a golden run. Catches routing regressions and unnecessary tool retries.

Layer 3 - Outcome evals. Test whether the final output is correct using LLM-as-judge or ground-truth labels. The most expensive layer. Run nightly, not on every commit.

Layer 4 - Online evals. Sample 1-5% of live production traffic and grade it asynchronously. The only layer that catches drift, prompt injection, and edge cases your golden dataset missed.

RAG Evaluation: Grade Retrieval and Generation Separately

If your agent answers from your documents, one accuracy score hides the real problem. A wrong answer can come from retrieval (the right passage never reached the model) or from generation (the passage was there and the model ignored or misread it). The fixes are completely different, so RAG evaluation has to score the two halves on their own.

In practice I build a set of 50-100 questions with the source passage labelled for each one, then run retrieval-only checks on every PR (they are cheap and need no judge model) and LLM-as-judge faithfulness checks nightly. Open-source libraries such as Ragas implement these metrics, and both LangSmith and Langfuse can store the scores next to the traces, so a failing answer links straight to the chunks it was given. For the retrieval patterns themselves, see agentic RAG patterns.

LangSmith vs Langfuse vs Braintrust vs Custom

LLM Evals in Practice: Agent Evaluation in LangSmith and Langfuse

The tool matters less than the workflow. Whichever you pick, running LLM evals for an agent comes down to the same four steps:

1. Build a dataset from real traffic. Pull 30-60 traced conversations - successes, failures, and edge cases - into a dataset. Both LangSmith and Langfuse let you promote production traces straight into datasets, which is how your eval set stays grounded in what users actually do.
2. Score the final answer. Attach an evaluator to each example: exact or structured match where the answer is deterministic, an LLM-as-judge rubric where it is not. This is classic LLM evaluation and catches most prompt regressions.
3. Score the trajectory. For agents, also check the path: which tools were called, in what order, how many times, and whether the agent stopped when it should. Store the expected tool sequence alongside each example and compare it to the traced steps. This is the layer that caught the double-call bug in the example below.
4. Run it on every change. Wire the eval run into CI so each prompt, model, or graph change produces a scored experiment you can compare against the last one, and sample a few percent of live traffic for online scoring.

LangSmith is the lowest-friction choice if you are already on LangChain or LangGraph: traces, datasets, and experiments live in one place with almost no setup. Langfuse does the same job framework-agnostically and can be self-hosted, which matters when trace data cannot leave your infrastructure. Either way, the dataset and the trajectory checks are the asset - the tool is replaceable.

What a Working Eval Stack Actually Catches

Real-World: Catching the $40K Regression

A B2B client running a customer-support agent on LangGraph started getting complaints about wrong answers in their billing flow. The team had no trajectory evals. They spent four engineering days bisecting recent prompt changes, found nothing, and were ready to roll back two weeks of work. We added a 60-prompt trajectory eval that took 90 minutes to wire up. The first run flagged that the billing-tool node was being called twice on 41% of trajectories - a routing change two weeks earlier had introduced a loop that doubled tool latency and bill, and occasionally returned stale data on the second call. We fixed the routing in 20 minutes. Total time from "we have a problem" to "we have a fix": 6 hours. Without trajectory evals it would have been three weeks.

5 Common Mistakes Teams Make

1. Evaluating on prompts that look like your training set. Real users send messy, malformed, ambiguous prompts. Your eval set must too. Pull eval prompts from production traffic, not from your imagination.

2. No golden dataset. You cannot measure regression without a baseline. Curate 50-100 prompts with verified correct trajectories before you ship v1, not after.

3. Only evaluating final outputs. The trajectory matters. A correct answer reached via 12 tool calls when it should take 3 is still a regression - it just costs 4x and fails the next time.

4. Running evals nowhere except locally. Evals belong in CI. If they do not block merges, they get skipped under deadline pressure.

5. No online sampling. Pre-deployment evals catch known failure modes. Online sampling catches the unknown ones - which is most of them in the first six months.

Frequently Asked Questions

Conclusion

You cannot ship an AI agent and hope for the best. The teams that win in 2026 are not the ones with the cleverest prompts; they are the ones with the eval stack that catches a regression in hours instead of weeks. Pick the layer you are missing, wire it up this week, and you will sleep better next month.

Need help wiring an eval stack onto an existing production agent? Happy to scope it in a free 30-minute call, or book an eval-coverage audit first.