Back

Tools & Frameworks

AI Evals

Effective AI evaluation starts with manually reviewing production traces — not with generic metrics dashboards. For agents specifically, the goal is usually floor-raising (eliminating failures that destroy trust) rather than benchmark-maxxing. Label ~100 traces by hand, categorize the failures, fix the obvious ones without an eval, then build targeted LLM-as-judge evals for the subjective problems that remain. Test the running agent path (tools, state, side effects), not isolated prompts. Every eval number needs two companions: an error bar (statistical rigor) and a grader accuracy metric (Cohen's kappa, not raw agreement). For agents, measure reliability (pass^k) separately from capability (pass@k) — two years of capability gains delivered roughly one-sixth as much reliability gain. Use specialized detectors and typed judges for narrow questions before reaching for expensive general LLM judges. Keep eval suites small, prune cases that haven't failed in months, and use benchmarks only to shortlist models — decide with your own eval on your own data.

Created May 28, 2026·Updated Oct 5, 2026

Recent Updates

Floor-Raising vs Benchmark-Maxxing

Success with AI products hinges on iterating fast across three activities: evaluating quality, debugging issues, and changing system behavior (prompts, fine-tuning, code). Most teams focus exclusively on changing behavior and plateau quickly — streamlining evaluation unlocks the other two.

Before designing evals, choose the frame. Benchmark-maxxing pushes top-end capability — useful when augmenting experts who can catch mistakes. Floor-raising eliminates the failures that destroy user trust — the right frame for agents that replace human workflows.

A finance agent that cross-references accounts and predicts spending is impressive. But if it sometimes can't answer "how much money is in my bank account?", users stop trusting it. Floor-raising addresses exactly this: making the agent reliable where reliability matters, in the workflows users actually run, in the moments where mistakes are expensive source(https://howtoeval.com).

The ROI evidence is mounting. Shopify used evals to guide an AI workflow builder that was 2.2x faster and 68% cheaper than the frontier-model system it replaced. Cursor developed its routing with evals, achieving higher user satisfaction at 41% lower cost. Ramp increased automatic receipt collection accuracy from 35% to 83%. Harvey nearly doubled its AI contract reviewer's internal quality score by rebuilding with evals source(https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix).

Floor-raising is error analysis. You're not starting with an abstract test suite — you're doing detective work. Finding the places where the system bends, classifying them, and deciding which ones deserve engineering effort. A floor-raising eval suite is a memory of bugs you refuse to reintroduce.

The litmus test: if you could ship with 90% or 99% pass rate, the benchmark-maxxer chooses 99%. The floor-raiser asks "which 1% fails?" source(https://howtoeval.com).

One of the lowest-hanging floor-raising techniques: teaching your agent to refuse. If confidence is low, say "I don't know." Benchmark-maxxers hate this — it lowers their pass rate. Floor-raisers understand that a confident wrong answer is worse than an honest refusal.

Golden Cases

Before trusting an agent in production, write down 5–10 cases that represent your critical paths. Start with the simplest version: a single common user question the agent should always handle correctly. If your agent starts failing golden cases, you do not ship source(https://howtoeval.com).

For each golden case, don't just check the final output — inspect the full trajectory: user message, tool calls, retrieved context, reasoning chain. The path matters as much as the answer.

Evals as Product Management Artifacts

At Anthropic, the research product management team operates under the principle that "evals are the new PRDs." The traditional PM artifact — the product requirements document — assumed deterministic software where specifying inputs and expected outputs was sufficient. With non-deterministic AI systems, the eval is the specification: it captures user need, defines success criteria, and provides the measurement loop that researchers use to improve the model.

The process starts with user pain. Rather than running traditional user interviews, PMs read model interaction transcripts — full trajectories of what the user asked, what the model did, and where it failed. The guiding principle: sweat the tokens as much as you sweat the pixels. Failure modes in AI products have nuance that surface-level feedback obscures — a user reporting "Claude doesn't follow instructions" might mean hallucination, overconfidence, schema non-compliance, or something else entirely. Only reading the actual token-level trajectory reveals the real category.

Once a failure pattern is identified, PMs generate 30–40 examples of the failure and assemble them into an eval set — each with a prompt, a model response, and the expected correct answer. This eval set becomes the durable artifact: it enters the repository, runs against every new model version, and tracks whether the pain point has been resolved. Early Claude models, for instance, struggled with JSON schema compliance — a pain point that surfaced from user complaints about "not following instructions." The resulting eval now consistently passes at ~100%, confirming the issue is resolved.

This is essentially test-driven development for product management. The PM writes the test (the eval) before the fix ships, and the test remains as a regression guard. Because AI outputs are non-deterministic, evals must describe success broadly rather than matching exact outputs — making eval design a skill that blends product judgment with technical specification. The approach extends beyond model teams: any product built at the intersection of models, harnesses, context, and users benefits from evals as the core quality loop, since "you can't improve what you can't measure" and measurement in AI products requires staying close to the failure details. For more on how this fits into broader AI-native product development, see the team-level workflow patterns there.

Three Levels of Evaluation

Evals form a cost pyramid. Level 1 (unit tests) runs cheaply on every code change. Level 2 (human and model-based eval) runs on a set cadence. Level 3 (A/B testing) runs only after significant product changes. There's no strict formula for when to introduce each level — balance getting user feedback quickly, managing user perception, and product goals. But conquering Level 1 before moving to Level 2 is important, since model-based tests require more work and time to execute.

The Trace Review Process

The foundation of good evals is looking at production traces — the full interaction logs between users and your AI system, including system prompts, tool calls, and internal reasoning steps.

Start with 100 traces. Review them one by one, writing short notes about what went wrong (or right). The first few are slow; by trace 10 you're fast. The goal is theoretical saturation — keep going until you stop learning new things. Most teams find 100 is more than enough.

Don't root-cause yet. During review, just journal what you observe. "The conversation dead-ended." "The AI offered virtual tours but there are none." "Markdown formatting in a text message." Don't debug, don't propose solutions — just note the problems.

Use AI to categorize, not to observe. Export your notes into a spreadsheet or CSV, then ask an LLM to group them into 5–6 categories (axial codes). Refine the categories until they feel right. Then use the LLM to classify each note into a category. A pivot table instantly shows which problems dominate.

Asking Your Agent

One underused debugging technique: reconstruct the run exactly as it was passed to the agent, along with the latest response, and ask it directly what happened. This works because reasoning traces are often opaque — passing the trace back to the same model is the closest you can get to understanding its reasoning, since it can use the available trace, context, and prior messages as evidence source(https://howtoeval.com).

Ask something like: "You were wrong. The answer was X. What would I need to have changed for you to get this right?" The answer isn't always the truth, but it surfaces where the agent is misinterpreting or overindexing on specific parts of the prompt.

Error Discovery

Error discovery is the eval equivalent of product discovery. Just as product discovery shows which problems are worth solving, error discovery reveals which AI failures are worth measuring. Most teams skip this step and jump straight to writing metrics — ending up measuring the wrong things or the right things poorly source(https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix).

The core insight: agents are good at catching failures obvious within a trace (e.g., answers contradicted by tool output) but miss issues requiring product judgment — like a leasing assistant that accepts "this is out of my budget" instead of offering cheaper alternatives. You'd only know to check for "objection handling" after seeing the trace yourself. This is criteria drift in action: grading outputs teaches you your criteria, but you need criteria to grade.

Hybrid Human + AI Annotation

The solution draws on active learning. Rather than handing an agent all traces and asking it to find problems, or reviewing everything manually:

  1. Start with a diverse sample. Cluster traces and mix cluster representatives with random picks to cover the range of your data.
  2. Review 10 traces by hand. Leave free-text notes on whatever bothers you — "The assistant gave up instead of offering alternatives," not "bad response." Stop at the first upstream error in each trace; upstream failures tend to be more important than downstream ones.
  3. Let the agent propose. After 10 annotations, the AI learns from your notes and suggests additional issues you can accept or reject. Don't blindly accept early proposals — use disagreements to correct its working understanding.
  4. Keep going to ~100 traces. Aim for theoretical saturation — keep reviewing until you stop learning new things. More traces also give the AI better signal.
  5. Cluster into failure modes. The AI groups your annotations into categories and counts them, producing a prioritized list of product issues you can act on.

Coding agents can scaffold this entire workflow. The evals-skills plugin builds a customized review app from your trace schema, renders conversations domain-appropriately (email as email, SMS as SMS), and handles the sampling and clustering bookkeeping. The biggest advantage: the interface is modifiable on the fly — if you notice a missing field or want to filter to a slice of traces, you ask the agent.

Why Error Discovery First

If you only have time for one part of the eval process, prioritize error discovery. Even without building any judges, manually annotating traces produces actionable insights. The annotation step is valuable on its own — many teams find obvious fixes during review that don't need an eval at all. The evals you do build afterward are grounded in real failures rather than assumptions about what might go wrong.

Two Types of Evals

Code-based evals (reference-based): When there's a clear right answer — dates, specific data lookups, format compliance — write deterministic assertions. These are cheap, fast, and don't need LLM judges. Fix the bug, write the test, move on. Many things are hard to produce but cheap to check once someone has done the preparation — the asymmetry of verification. If you have an expected output, every future evaluation run becomes a cheap deterministic check rather than an expensive LLM judgment source(https://langfuse.com/academy/evaluate/writing-evaluators).

Organize assertions by feature and scenario. For a real estate listing finder: "only one listing matches query" → assert len(results) == 1; "multiple matches" → assert len(results) > 1; "no matches" → assert len(results) == 0. Generic assertions (e.g., regex-checking that UUIDs aren't leaked in user-facing output) apply across all features source(https://hamel.dev/blog/posts/evals/). Hundreds of these tests, continuously updated as new failure modes appear in production data, provide rapid feedback when iterating on prompts or RAG.

Unlike traditional unit tests, a 100% pass rate isn't always required — the acceptable pass rate is a product decision depending on which failures you can tolerate.

LLM-as-judge evals (reference-free): For subjective quality — conversational flow, appropriate handoffs, tone — prompt an LLM to judge traces against specific criteria. These are more expensive but provide the most value for iterating on hard problems.

Decision-model evals (emerging third type): System One models like TypeSafe's Jev occupy a middle ground — they evaluate typed questions against structured state and return typed answers with probabilities, without generating text. This makes them faster and cheaper than LLM judges while handling the unstructured input that code-based evals cannot. Early benchmarks show dramatically lower variance than LLM judges on the same tasks, though the category is still narrow in scope source(https://x.com/LangChainAI/status/1934295107724808620).

Not everything needs an eval. If the fix is obvious (e.g., the model doesn't know today's date because you forgot to include it), just fix it. Evals earn their keep on problems you'll iterate against repeatedly.

Specialized Detectors

For the failure types that matter most — particularly hallucination — a general LLM judge is often the wrong tool. Small models (100M–500M parameters) trained for one question ("is this sentence supported by this context?") run on a laptop CPU in about a second, cost nothing per call, and approach the accuracy of large judges on supported-claim detection.

Hallucination detectors: Vectara's HHEM, LettuceDetect (Kovács & Recski, 2025), MiniCheck (Tang et al., 2024), and plain NLI cross-encoders all target the same narrow question. SelfCheckGPT (Manakul et al., 2023) needs no detector at all — it samples the system several times and flags claims that change between samples, exploiting the observation that hallucinated content varies while supported content remains stable source(https://makarevych.substack.com/).

The cost difference matters at production scale. Teams running a general LLM judge on every production trace typically sample only ~10% of traffic because of cost. Small detectors can cover 100% of traffic for specific failure types, catching regressions that sampling misses.

Typed judges like TypeSafe's Jev occupy a middle ground — they return calibrated probabilities for typed questions without generating text, making them ~100x cheaper than generative judges. A 2026 blinded human-adjudication study put Jev at 92.2% on RewardBench pairs versus 93.5% for a frontier generative judge, at ~0.3% of the cost. On hard derivation-checking, however, the gap widened (78.6% vs 93.1%). A confidence-threshold cascade — accepting Jev's verdict only above a threshold and escalating otherwise — maintained 92.5% accuracy at roughly half the cost of the generative judge alone source(https://makarevych.substack.com/).

The pattern: use code rules for definite answers, specialized detectors for "is this supported," typed judges for narrow classification, and general LLM judges only for the subjective criteria that simpler tools cannot handle.

Code-Aware Offline Evals

Testing prompts in isolation makes no sense once the agent is entangled with code, tools, retrieval, permissions, and product state. The behavior lives in the whole system, not in the prompt string. Offline evals should look less like prompt scoring and more like ordinary software testing — Vitest, pytest, or jest source(https://howtoeval.com).

A good offline eval takes an input, runs the real agent path, and asserts on the result: output, tool calls, files changed, structured data, or final state. Sentry's vitest-evals demonstrates this with describeEval(...), an app-local harness, an explicit run(...), normal expect(...) assertions, and tool-call checks. OpenAI calls the same idea "macro evals" in their agentic systems cookbook: drive the real agent loop on representative inputs and grade the full trajectory source(https://howtoeval.com).

The key principle: evaluate the agent, not an LLM call.

Containerized Evals (Harbor)

The Harbor framework formalizes containerized evals as three components: an instruction (the task prompt given to the agent), an environment (a Dockerfile that installs tools, populates data, and sets up the filesystem), and a verifier that scores whether the agent completed the task correctly. Harbor runs the agent inside the environment and records its trajectory, artifacts, reward, and errors source(https://x.com/vtrivedy10/status/2079976006644072796/).

Containerized environments make the eval loop faster. The task and environment remain stable while the agent configuration changes — builders can swap models, tools, prompts, or complete agent versions and compare results directly. Multiple configurations can run in parallel. Reproducible environments are critical to signal quality: when an eval mirrors the relevant tools, data, permissions, state, and failure modes from production, builders get a stable testbed that is still representative of how the agent operates.

Agent Reliability

Agent evaluation requires dimensions beyond pass/fail. Two numbers describe the same result differently: pass@k (succeeded at least once in k tries — measures capability) and pass^k (succeeded all k times — measures reliability). For customer-facing agents, pass^k is the right metric. τ²-bench (2025) found that pass^1 drops ~20 points when the agent must coordinate with an active user rather than act alone source(https://makarevych.substack.com/).

Reliability as its own axis. Rabanser et al. (2026) borrowed from aviation and nuclear engineering to define 12 reliability metrics across four groups: consistency across runs, robustness to perturbation, predictability (does the agent know when it will fail), and bounded harm. Across 15 frontier models they found that two years of capability gains delivered roughly one-sixth as much reliability gain. Agents choose the same kinds of actions each run but in different orders, handle real infrastructure failures gracefully, and break on a rephrased instruction. A capable agent is not a reliable one, and customers experience reliability source(https://makarevych.substack.com/).

Cost measurement. Bai et al. (2026) measured token spend across eight frontier models on SWE-bench Verified. Agentic tasks consume roughly 1,000x more tokens than a chat turn — almost all of it re-reading growing history. The same task on the same model varied 30-fold in cost between runs, accuracy peaked at intermediate cost and fell at the highest, and models predicted their own cost with a correlation of at most 0.39. Rank models on cost per correct outcome, not accuracy alone source(https://makarevych.substack.com/).

Judgment, asking, and learning. Three 2026 benchmarks measure things a single success rate cannot see. Taste (Pan et al.): freeze an agent at a fork where both paths look fine and ask which pays off later — the best model picks right 59.7% of the time, and thinking longer doesn't help. Asking (Gulati et al.): the value of a clarifying question drops from 0.78 to 0.39 once 10% of the task is done — no frontier model asks within that window. Learning from experience (Asawa et al.): the best system captured only 25% of its possible improvement from experience, and keeping the full conversation history beat every dedicated memory product. These three axes — taste, timely clarification, and experiential learning — are emerging as the frontier dimensions for agentic engineering evaluation source(https://makarevych.substack.com/).

Reward Hacking

When iterating on verifiers, watch for reward hacking — agents taking shortcuts that satisfy the verifier without completing the task. Common patterns: overciting irrelevant sources to receive full credit, claiming an action never taken, exploiting exposed answer material, or satisfying a proxy metric without actually solving the problem source(https://x.com/vtrivedy10/status/2079976006644072796/).

The fix is to inspect both sides of the result: the agent trajectory (messages, tool calls, actions) and the verifier trajectory (evidence, reasoning, final score). This dual inspection reveals whether the task or verifier design is actually measuring what you care about. The first verifier is rarely the final one — the task, environment, and verifier typically go through several revision cycles.

Verification Horizon

Wang et al. (Qwen, 2026) formalize the deeper problem as the verification horizon: every grader is a proxy for intent, and under optimization pressure proxies and intent diverge. No grader is simultaneously scalable, faithful, and robust — tests are scalable and robust but miss intent; judges are scalable and faithful but gameable; human experts are faithful and robust but don't scale. In coding agents specifically, they catalogued hacking behaviors including reading answers from repository history, editing the tests, and patching for the evaluator. A behavior monitor cut hacked "solutions" from 28.6% to 0.6% while raising genuine solutions from 40% to 61%. The practical defense: keep a held-out grader the system never sees, monitor for shortcut behavior, and upgrade the grader as the system improves source(https://makarevych.substack.com/).

Automated Eval Engineering

Coding agents can automate the eval-authoring process itself. LangChain's Eval Engineering Skill demonstrates this: the skill inspects how an agent is structured (prompts, models, tools, hooks), mines patterns from production traces (via tools like langsmith-cli), and proposes abilities to test. Crucially, the skill interviews the user rather than generating evals in one shot — users choose from proposed eval directions and specify which tools should run live versus be simulated (e.g., tool calls that incur costs or write to production) source(https://x.com/vtrivedy10/status/2079976006644072796/).

The resulting loop is: mine traces → identify a failure → build an eval → improve the agent → rerun. This aligns with the view that continual learning is a continuous data-mining problem where production data feeds evals that improve agents over time.

Building an LLM Judge

Write a judge prompt that specifies the exact criteria for pass/fail. For a human-handoff judge: list the specific failure modes (user requested transfer but was ignored, too many loops before handoff, etc.) and the conditions under which there is no failure.

Label real cases before writing the prompt. Evaluation criteria should not come exclusively from your head. Criteria drift is the catch-22: you need criteria to grade outputs, but grading outputs is what teaches you your criteria source(https://langfuse.com/academy/evaluate/writing-evaluators). Take 10–20 real cases of the failure mode you want to evaluate and label each one with a short comment. That's enough to build a robust judge. If you already did error analysis, most of this exists already.

Write the prompt like onboarding material. The bar: a new colleague could read it and reach the same verdicts you would source(https://hamel.dev/blog/posts/llm-judge/). A judge prompt has five parts source(https://langfuse.com/academy/evaluate/writing-evaluators):

  1. Context — what the application does and the domain knowledge needed to check the criterion.
  2. A precise criterion, including what to ignore — "Is the response high quality?" is a question two people would answer differently. "The response cites at least one source document. Ignore formatting issues." gets you the same answer every time.
  3. Labeled examples with reasons (optional) — 2–4 of your labeled cases mixing pass and fail. Start without them; add only when the judge isn't accurate enough, since they increase token consumption.
  4. Reasoning first, verdict last — reasoning-first prompting measurably improves judge accuracy source(https://eugeneyan.com/writing/llm-evaluators/). The reasoning is also the first thing you read when you disagree with a verdict.
  5. An explicit way out — let the judge answer "unknown" when information is missing instead of guessing.

Always output binary or categorical. Continuous scales (1–5) introduce enormous complexity. LLMs are poor at consistent numeric scoring — they even develop favorite numbers (GPT-3.5 has a documented preference for the number 7) — the results aren't actionable ("3.2 vs 3.7 — is that better?"), and they create false precision that erodes stakeholder trust. A pass/fail verdict is easily verifiable: you can count exactly how often the evaluator catches a failure and how often it clears a pass. There is no equivalent test for whether a 7 was the right score. When one event has several mutually exclusive outcomes, use a single categorical evaluator that picks one label per case (resolved / abandoned / handed_off) rather than several overlapping binary ones source(https://langfuse.com/academy/evaluate/writing-evaluators). If you can't define a clear pass/fail threshold, you probably don't understand the problem well enough yet.

Add explanations alongside the score. Have the judge output a structured response with both an explanation field and a binary score. The explanation helps you debug the judge's reasoning when it disagrees with your labels.

Measuring Judge Quality

The critical step most teams skip: validating that the judge actually works. You already have human labels from the trace review — use them as ground truth.

Never use raw agreement. If a failure occurs 10% of the time, a judge that always says "no failure" achieves 90% agreement. That looks great on a dashboard and is completely useless.

Use true positive rate and true negative rate instead. TPR: how often does the judge correctly identify real failures? TNR: how often does it correctly identify non-failures? A confusion matrix makes this concrete — you can see exactly where the judge is wrong and in which direction. (Note: raw agreement can be appropriate when failure classes are roughly balanced, around 50/50 — but this is rare in production systems.)

Cohen's kappa corrects for chance agreement: zero is random, 0.6 is the usual bar for a judge you let gate a release, and 0.8 for one that runs unsupervised. A bike-shop support eval found 60% raw agreement between a judge and human labels — which sounds acceptable until you calculate kappa at 0.15, because the judge simply said "pass" more than it should have. Raw agreement was only 3 points above what always-pass would achieve. The gap between raw agreement and kappa is the single fastest diagnostic for a lenient judge source(https://makarevych.substack.com/).

Satisfaction is not success. The GAUGE audit (Bodhwani et al., 2026) tested the exact setup most companies use: simulated user, agent transcript, judge score, higher score ships. Across 25 agents and ~3,700 transcripts, a judge that only saw user-visible chat and rated satisfaction was no better than a coin flip at distinguishing working agents from broken ones (AUC 0.49). A judge given tool calls, authentication steps, and task resolution state halved the failure risk (AUC 0.73). On near-equal pairs — the ones a real release actually decides between — the satisfaction-only judge promoted the worse agent 31% of the time. Reliability comes from what evidence the judge receives, not from how strong the judge model is source(https://makarevych.substack.com/).

One practical alignment workflow: send a domain expert a spreadsheet with the model's response, the critique model's written critique, and the critique model's binary label. The expert fills in their own critique, label, and preferred response for 25–50 examples at a time. Iterate on the critique prompt until alignment stabilizes, then continue periodic spot-checks to monitor drift source(https://hamel.dev/blog/posts/evals/).

Whether the error rates are acceptable is a business decision, not a statistical one. False positives (flagging non-failures) are usually cheaper than false negatives (missing real failures). Look at the specific misclassified traces, iterate on the judge prompt, and re-measure.

Hold out data when adding examples. If you put few-shot examples in the judge prompt, don't test against those same examples — you'll get artificially perfect scores.

Statistical Rigor

Every eval rate hides a question: would it be the same on a different set of cases? Most reported differences between models in published evals are smaller than the confidence interval the eval can actually detect (Miller, 2024).

Error bars. Bootstrap-resample your results and report the spread. "Accuracy is 70%" is a point estimate; the honest statement is "somewhere between 58% and 82%, most likely 70%" (for a 60-case test set with a 95% CI). Report the interval alongside every headline number.

Paired comparisons. When comparing two versions, run both on the same inputs. The confidence interval narrows because per-case difficulty cancels out. But headline "same pass rate" can mask a system that got better for some users and worse for others — always report the count of cases that regressed alongside the aggregate improvement source(https://makarevych.substack.com/).

Detection floor. With 60 cases, the smallest difference you can reliably detect is roughly 12 percentage points. A 5-point regression is invisible at this size. A power calculation tells you how many cases you need before you can answer the question you're asking. Below the floor, you need more data, not a cleverer test source(https://makarevych.substack.com/).

Multiple comparisons. If you test 20 prompt variants and pick the best, one of them will look 10 points better by luck. Either correct for the number of comparisons or confirm the winner on fresh cases.

Four numbers for a leader. The product's pass rate with its interval, the grader's agreement with human labels, the detection floor, and the share of cases that got worse in the last change. If any of the four is missing, the other three are not yet meaningful source(https://makarevych.substack.com/).

Runtime Judges

Most LLM-as-judge usage today sits outside the application execution path — offline evaluation against datasets. An emerging architectural shift moves judges inside the agent loop, where judgment informs control flow: continue, retry, route to another model, gather more evidence, or escalate to a human source(https://x.com/joshuacrosen/status/1929256835976724863). The judge is no longer just telling you whether the application worked yesterday — it's helping determine what the application does next.

This is a much bigger role. Mistakes from an offline judge are annoying; mistakes from a runtime judge become application failures — creating loops, blocking good work, approving bad work, and adding latency to every execution.

Specialized Judge Models

The runtime judge doesn't need to be your strongest frontier model. Judging a narrow property (e.g., "are all claims supported by the supplied evidence?") can be far easier than producing the original work. Specialized judge models like Prometheus and distilled evaluators trained for particular judgment types can make runtime judging economically viable at scale. Galileo has built evaluation models specifically designed to be cheap enough for continuous production-scale use source(https://x.com/joshuacrosen/status/1929256835976724863).

A more radical departure: System One decision models like TypeSafe's Jev skip text generation entirely. Jev evaluates a state — the agent's trace, context, and outputs — against typed questions (choice, score, or boolean) and returns calibrated probabilities directly. LangChain benchmarked Jev against GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 on a weather-agent eval set (five cases, 100 repetitions each, human-labeled oracle). On binary pass/fail accuracy, Jev matched the oracle on all 500 decisions (100%); Terra reached 99.8%, Luna 96.4%, Claude 80.0%. On continuous quality scoring, Jev's mean per-case variance was 0.0000149 — 92x lower than Claude, 433x lower than Luna, and 913x lower than Terra. Average latency was 0.44s at $0.00035/call, totaling $0.34 for the full run versus $28.17 for Claude source(https://x.com/LangChainAI/status/1934295107724808620).

The benchmark is narrow (five weather queries, one agent), and low cost can amplify mistakes if the judge is consistently wrong. But the direction is significant: if decision-first architectures maintain these accuracy and variance properties across broader agent behaviors, they could make online evaluation economically feasible at a scale that LLM judges cannot reach.

Decomposed Judgment

Rather than asking one model to make a single fuzzy "was this good?" decision, break the judgment into smaller decisions with well-scoped prompts and put structure around how they combine. G-Eval pioneered this by having the model generate evaluation steps from the criteria before scoring. DeepEval extends it with DAG-based evaluation — individual LLM judgments sit inside a larger deterministic decision graph source(https://x.com/joshuacrosen/status/1929256835976724863). This aligns with the anti-pattern advice against "God Evaluators": narrow judges checking one criterion each are easier to build, calibrate, and debug.

Pairwise Comparison

Models are better at deciding which of two outputs is better than assigning absolute scores. Pairwise judging takes advantage of this — given two outputs from the same input, the judge picks which better satisfies the criteria. LangSmith and DeepEval support this directly, including answer-order randomization to reduce position bias source(https://x.com/joshuacrosen/status/1929256835976724863).

At runtime, this enables architectures where an agent generates multiple plans and a judge selects the best, or two agents independently analyze a problem and a comparator picks the stronger result. A proposed action can be compared against an alternative before the system commits.

Multi-Judge Panels

A single judge is still an LLM — it can make mistakes for the same reasons the worker model can. Multi-judge architectures use panels, voting, and deliberation to increase confidence. MAJ-EVAL creates multiple evaluator agents representing different evaluation dimensions and lets them deliberate source(https://x.com/joshuacrosen/status/1929256835976724863).

The practical value for runtime architectures isn't necessarily the vote outcome — it's the disagreement signal. Agreement between independent judgments increases confidence; disagreement triggers retry with a stronger model, additional evidence gathering, or human escalation. This maps naturally to the retry and routing logic that runtime agents already need.

Panel combination has a statistical trap: Acharya et al. (2026) proved that one broken judge (a parser failure returning zeros, a sycophantic judge rating everything 10) drags the average arbitrarily far off, no matter how many good judges you add. Real judges fail at measurable rates — 3.4% parser failures on one dataset, up to 33% on the smallest judge for non-English prompts. Their fix: replace the average with the geometric median, which ignores outliers. In the worst case it was 540x more accurate; on clean data it costs ~1% accuracy. They also show three judges capture most of the benefit, because judges' errors are correlated. A related finding: Parikh (2026) found that asking a model for answers in JSON (the format every judge pipeline uses) makes its responses measurably more uniform — the modal answer's share rose from 41% to 64% — which means a JSON-mode judge may be less discriminating than its chat behavior suggests source(https://makarevych.substack.com/).

Meta-Evaluation

Once a judge affects application behavior, its reliability must be measured. LLM judges exhibit known biases: position bias (changing decisions based on answer order), style preference, difficulty distinguishing close-quality outputs, and self-preference for outputs resembling their own. Benchmarks for measuring these problems include LLMBar (instruction-following under adversarial conditions), JudgeBench (difficult response pairs with objective preferences), and RewardBench (reward models across challenging preference tasks) source(https://x.com/joshuacrosen/status/1929256835976724863).

Anthropic's Bloom project provides an accessible pattern: evaluate candidate judge models against human-labeled transcripts before choosing the production judge, then add a meta-judge that reviews evaluation results at a higher level. The simpler version any team can adopt today: periodically compare the judge against humans on a sample of production decisions and adjust the rubric, model, or decision boundary when disagreement is unacceptable.

Hybrid Deterministic + LLM Checks

The biggest advantage at the application layer is controlling the surrounding system. If something can be checked deterministically — schema validation, type checks, database lookups — check it deterministically. OpenAI's grader architecture formalizes this separation with model-based graders alongside deterministic graders (string checks, Python code), combinable into larger evaluations source(https://x.com/joshuacrosen/status/1929256835976724863).

Inside runtime agents, an LLM might judge whether evidence is sufficient or a recommendation well-supported. Deterministic code decides what combination of those judgments and other facts is required before the workflow proceeds. Drawing this boundary well — knowing where LLM judgment adds value and where determinism suffices — is one of the highest-leverage architectural decisions for agent builders.

Benchmark Limitations

Public benchmarks (MMLU, GSM8K, SWE-bench, Chatbot Arena) are how you pick an engine, but four properties limit what they tell you.

Harness dependency. A score is a (model, prompt, harness, version) tuple — not a property of the model alone. Running the same open-weight model with two prompt formats on GSM8K yielded 0.686 and 0.746. The lm-evaluation-harness paper (Biderman et al., 2024) documents double-digit swings from prompt formatting, few-shot examples, and answer-extraction regex. Schaeffer et al. (2023) showed that apparent "emergent" capability jumps were largely artifacts of all-or-nothing metrics and vanished under smooth ones — the metric itself can create or erase a phenomenon source(https://makarevych.substack.com/).

Contamination. If the test questions were in the training data, the score measures memory. GSM1k (Zhang et al., 2024) rebuilt GSM8K with fresh questions of matched difficulty; some model families dropped up to 8 points, with the drop correlated with memorization likelihood. Contamination is real, measurable, and uneven across model families source(https://makarevych.substack.com/).

Saturation. Benchmarks have a shelf life. A 2026 study of 60 widely used benchmarks found about half highly saturated; older ones saturate faster. SWE-bench Verified went from ~40% to over 80% solved in about two years. On Chatbot Arena the top models sit within ~20 Elo points — inside the noise. MMLU, GSM8K, and HumanEval have been effectively retired by frontier labs in favor of HLE, GPQA Diamond, LiveCodeBench, and τ²-bench source(https://makarevych.substack.com/).

Redundancy. Zeng & Papailiopoulos (2026) assembled 84 models × 133 benchmarks and found it is approximately rank two: two underlying factors explain over 90% of the variation. Five well-chosen benchmarks predict the other 128 to within ~4 points. For model selection, that's good news. For discovering a specific failure mode, it says nothing — that still needs your own eval source(https://makarevych.substack.com/).

The practical advice: use benchmarks to shortlist two or three models, then decide with your own eval on your own data. Switching models on a 2-point benchmark gain can cost you 10 points on your actual task.

Removing Friction from Trace Review

The biggest bottleneck in eval quality is friction in the data-review loop. Render traces in domain-specific ways that put all relevant context on one screen — the trace log, the CRM state, tool outputs, and links to upstream systems. Off-the-shelf logging tools (LangSmith, etc.) are a starting point, but custom viewing and labeling tools built with lightweight frameworks (Gradio, Streamlit, Shiny) in less than a day often outperform them for domain-specific needs source(https://hamel.dev/blog/posts/evals/).

Design choices that compound: make the final LLM output editable by a human so reviewed traces double as curated fine-tuning data. Add filters by feature and scenario. Embed the downstream application view so reviewers see exactly what the user saw.

Start with binary labels (good/bad) rather than granular scores — they're simpler to manage and less onerous for labelers. More advanced techniques (active learning, consensus voting) can come later.

Production Monitoring

Once a judge is validated, deploy it against production traces:

  • CI integration: Run judges on test cases whenever you change code or prompts.
  • Sampling in production: Score a random sample of live traces to monitor whether failure rates are trending up or down. When comparing judges for production monitoring, signal value — oracle agreement × repeatability (the chance two independent calls on the same trace return the same verdict) — rewards judges that are both accurate and stable while penalizing those that are consistently wrong source(https://x.com/LangChainAI/status/1934295107724808620).
  • Debugging at scale: Use judges to find all instances of a specific failure type across production data, then drill into those traces.

Keep the eval suite small — typically under a dozen judges. Each one has maintenance cost (periodic re-labeling, prompt updates as the product evolves). Code-based evals are cheaper to maintain than LLM judges.

Scaling Review with Volume

The workflow should scale with traffic. At ten agent runs a day, read everything. At ten thousand, the system needs to tell you what deserves attention. The mistake is reaching for high-volume machinery before you have enough raw texture to know what you're looking for source(https://howtoeval.com).

Start with raw logs as the firehose — look for confusion, frustration, near-misses, repeated prompts, and "you're holding it wrong" moments. When the same stumble keeps appearing, promote it to an issue. Track long-horizon behavioral signals (refusal quality, ignored tool errors, context loss, user frustration). Once you understand an issue, ship the fix and compare affected metrics. Production tells you whether the change actually helped.

Eval Suite Hygiene

Not every bug deserves an eval case. Ask: is this a critical path? Could it regress? Is it representative of a class of failures, or truly one-off? Be ruthless about pruning — 20 high-signal cases beats 200 low-signal ones source(https://howtoeval.com).

A good heuristic: if an eval case hasn't failed in 3 months, it's either not testing something important or the agent has genuinely improved. Either way, question whether it needs to be there. The pattern for non-trivial issues remains: find in production, reproduce locally, add to evals, fix, verify, ship.

Plan to spend 10–20% of agent development time on evaluation and monitoring — not just writing eval cases, but reading traces, tuning signals, and investigating issues source(https://howtoeval.com).

Synthetic Data for Pre-Launch

Before you have real users, generate synthetic test inputs by defining dimensions — the axes of variation that matter for your product. For a property management assistant: customer type (resident vs. manager), property class (luxury vs. standard), interaction type (maintenance vs. leasing). Generate the cross-product, then ask an LLM to create plausible queries for each combination — using a separate model call per scenario to maximize diversity. This explores the input space far more effectively than asking an LLM to "generate some test questions" source(https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix). For more on generating realistic diverse user cohorts — including evolutionary optimization of persona-generating code — see Synthetic Personas.

Synthetic generation also works for unit test cases. Prompt an LLM to produce N variations of realistic user inputs for a specific feature, then pair each with a verification query. For a CRM contact creator: generate "Create a contact for John Smith (johndoe@apple.com)" paired with "What's the email address of John Smith?" — then assert the round-trip succeeds. One signal that your tests are good: when the model struggles to pass them, those failure modes become tractable targets for fine-tuning source(https://hamel.dev/blog/posts/evals/).

Eval Infrastructure Reuse

A well-built eval system generates two additional superpowers nearly for free:

Fine-tuning data. 99% of the labor in fine-tuning is assembling high-quality data with good coverage. If you have eval infrastructure, you already have synthetic data generators (reuse your test-case prompts), quality filters (Level 1 assertions and Level 2 critique models reject bad examples), and human-curated traces from the labeling tool. Fine-tuning is best for learning syntax, style, and rules; RAG is better for supplying context and up-to-date facts.

Debugging. When you get a complaint or observe an error, a robust eval system gives you a searchable trace database, assertion mechanisms that flag known bad behaviors, log navigation tools for root-cause analysis (RAG failure? code bug? model limitation?), and the ability to make a fix and quickly test its efficacy.

The Collapse of Harnesses

The current model of "agent SDKs that call models" is showing cracks. As models become more capable and agentic, the distinction between "the model" and "the agent" blurs — visible in Claude Code, the Cursor CLI, and emerging agent SDKs. When the agent is just a prompt and a model, with no framework code to instrument and no explicit tool loop to trace, the harness collapses into the model itself source(https://howtoeval.com).

This changes monitoring. Today we trace tool calls because we can. Tomorrow we might record input/output pairs and ask the model to explain what it did — like asking a human what they were thinking. Verification becomes harder.

The implication: end-to-end evaluation becomes even more important. If you cannot inspect the internals, you have to trust and verify the outputs. Golden cases and production monitoring become the only game in town.

Common Anti-Patterns

  • "God Evaluator": A single judge that rates accuracy, tone, and completeness together on a 1–10 scale source(https://langfuse.com/academy/evaluate/writing-evaluators). The resulting score doesn't tell you what to fix. Narrow judges that each check one criterion are also easier to build — getting a judge to agree with you on one specific criterion is a much lower bar than agreeing on "quality" source(https://eugeneyan.com/writing/product-evals/).
  • Generic metrics first: Starting with helpfulness/toxicity/coherence dashboards instead of looking at actual traces. These generic scores rarely match the real problems in your application.
  • Skipping annotation: Jumping straight to automated evals without manually reviewing traces. The annotation step is the single most valuable part of the process — even without building any judges, it produces actionable insights.
  • Continuous scoring: Using 1–5 scales instead of binary pass/fail. Adds exponential complexity for marginal benefit. Teams must be "extremely disciplined" to use Likert scales correctly, and most aren't.
  • Reporting raw agreement: Using agreement percentage instead of TPR/TNR. Misleading whenever failures are rare (which they usually are in a working system).
  • Over-investing pre-launch: Building elaborate eval suites before you have real users. Dog-food the product, get a few friendly customers, and save the serious eval infrastructure for when you have enough data that manual review can't keep up.
  • Focusing only on changing behavior: Jumping straight to prompt engineering or fine-tuning without investing in evaluation and debugging infrastructure. This is the most common root cause of AI products that never progress beyond a demo.
  • Massive eval suites: Adding every bug as an eval case without pruning. Six months later you have 500 cases, CI takes 20 minutes, and the team starts ignoring failures because "it's always something" source(https://howtoeval.com).

The Advanced Use for Generic Scores

Generic scores (hallucination, coherence, etc.) have one legitimate use: as a sampling mechanism. Sort your traces by a generic score, then manually review the highest-scoring ones to see if the score correlates with anything real. If it does, you've found a useful filter for prioritizing trace review. But never report generic scores directly to stakeholders.

Shadow Testing

Shadow testing (also called shadow runs) lets a candidate agent version process real production traffic while its output is shown to nobody. The candidate runs side-by-side with the live agent; only the live agent's responses reach users. This de-risks deployments by revealing failures under real-world input distributions before any user sees the new version. Shadow runs sit between offline evals and full A/B tests — they use real traffic (unlike offline) but carry zero user-facing risk (unlike A/B). Langfuse supports tracing shadow runs, comparing candidate outputs against the live baseline, and monitoring eval results across both.

Red Teaming

Red teaming attacks the system before an adversary does — probing for jailbreaks, prompt injection, data leaks, and tool abuse. Unlike other eval types that measure whether the agent does the right thing, red teaming measures whether the agent can be made to do the wrong thing. Garak scans LLM systems for vulnerabilities and unsafe behavior, automating adversarial probes that would otherwise require manual security expertise.

Eval Tooling Landscape

A non-exhaustive map of OSS tools to eval categories covered on this page:

  • Golden sets / regression suites: Promptfoo — repeatable eval suites with CI integration; catch regressions across prompt, model, or toolset changes
  • LLM-as-judge: OpenEvals — ready-made evaluator templates for LLM applications
  • Multi-dimensional scoring: DeepEval — custom metrics with independent scoring per dimension (correctness, tone, safety, cost)
  • Trajectory eval: AgentEvals — grades agent actions, decisions, and tool calls across the full run trajectory
  • Tool unit testing: MCP Inspector — inspect and test MCP server tools and responses in isolation, no model in the loop
  • A/B testing: GrowthBook — feature flags, controlled experiments, and product analytics for splitting real traffic between agent versions
  • Error discovery: evals-skills — coding agent plugin that scaffolds hybrid human+AI trace annotation; builds customized review apps, clusters traces, and guides the annotation workflow
  • Human review: Argilla — collect human feedback, review outputs, and build labeled datasets for judge calibration
  • Shadow testing: Langfuse — trace production and shadow runs, compare candidates, monitor eval results
  • Decision-model judge: Jev — TypeSafe AI's System One evaluator; typed answers with probabilities instead of text generation; ultra-low variance and cost
  • Red teaming: Garak — automated adversarial scanning for LLM vulnerabilities

Sources

  • Advanced evals: How to find (and fix) hidden AI failures in your product | Hamel Husain & Shreya Shankar (Lenny's Newsletter) — error discovery as the eval equivalent of product discovery; hybrid human+AI annotation workflow; criteria drift in practice; coding agent evals-skills plugin; company ROI evidence (Shopify, Cursor, Ramp, Harvey); synthetic trace generation with dimensions
  • AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain (Peter Yang, video) — foundational source; end-to-end walkthrough of the trace-review-to-LLM-judge pipeline using Nurture Boss (AI property management assistant) as a real-world case study
  • Your AI Product Needs Evals | Hamel Husain — three-level eval framework (unit tests → human/model eval → A/B testing); Rechat/Lucy case study; assertion-based unit tests; synthetic test generation; custom trace-viewing tools; eval infrastructure reuse for fine-tuning and debugging
  • How to evaluate AI agents | howtoeval.com — floor-raising vs benchmark-maxxing frame; golden cases; code-aware offline evals (test the agent, not the LLM); asking your agent directly; eval suite pruning; production monitoring at scale; collapse of harnesses
  • Towards Automating Eval Engineering | Viv (LangChain) — Eval Engineering Skill for coding agents; Harbor containerized eval format (instruction + Dockerfile + verifier); interview-driven eval design; reward hacking patterns; continual-learning-as-data-mining loop
  • 10 agent evals every AI engineer should know | elune — beginner-level overview of 10 eval categories with OSS tool recommendations; shadow testing and red teaming concepts; specific tooling landscape
  • Good evals are boring | Lotte (Langfuse Academy) — practical eval methodology: asymmetry of verification; "God Evaluator" anti-pattern; criteria drift; five-part judge prompt structure; categorical scoring; labeled-case calibration workflow
  • Evals are the new PRDs: how AI is rewriting the PM job | Dianne Penn (Lenny's Podcast, video) — Anthropic's PM workflow where evals replace PRDs; user-feedback-to-eval pipeline; "sweat the tokens" principle; JSON schema compliance as origin case study; test-driven development analogy for PM work
  • LLM-as-Judge Architectures: Putting Evals Into Your Agent Runtime | Josh Rosen — runtime judge architectures: specialized judge models, decomposed judgment (G-Eval, DAG-based), pairwise comparison, trace-level judging, multi-judge panels (MAJ-EVAL), meta-evaluation benchmarks (LLMBar, JudgeBench, RewardBench, Bloom), hybrid deterministic+LLM control flow (OpenAI grader architecture)
  • Jev-as-a-Judge for Agent Evals | Daniel Shea & Seán Roche (LangChain) — benchmark of TypeSafe Jev "System One" model vs GPT-5.6 and Claude Sonnet 4.6 as agent judges; 92–913x lower variance; $0.00035/call; signal-value metric; decision-model evaluators as emerging third category
  • Evals: how to know whether an AI system actually works | Sergii Makarevych — comprehensive level-by-level guide: statistical rigor for eval numbers (error bars, detection floor, paired comparisons); Cohen's kappa for judge validation; GAUGE audit on satisfaction vs success; hallucination detectors (HHEM, LettuceDetect, MiniCheck, SelfCheckGPT); agent reliability (pass^k, Rabanser's 12 metrics, cost measurement); verification horizon (Qwen); benchmark limitations (contamination, saturation, redundancy); RoPoLL geometric median for judge panels; judge bias catalog with mitigations