Evals for AI Agents: Making Quality Measurable
Many AI agents look good in a demo. The trouble starts in week three, when someone notices that a share of support tickets were closed with the wrong answer and nobody can say how many. Evals replace "it seems to work" with a number you can track across versions, compare between models and write into a sign-off. This guide covers what to measure, who or what does the grading, and how to turn results into a release decision.
What are evals and why do teams use them?
Evals are repeatable, systematic tests of how a language model or AI agent behaves. A fixed set of tasks with clear success criteria is run automatically and scored. The output is a metric you can use to compare versions, catch regressions and decide whether an agent is ready for production.
Traditional software tests assume the same input always produces the same output. Language models break that assumption: the same customer email can be handled correctly on one run and badly on the next. Evals deal with this by running many cases several times and reporting rates rather than single results. For a short definition, see the glossary entry on AI evals.
Teams typically use evals for four decisions:
- Model selection: which model handles your tasks reliably enough, at what cost?
- Change control: did the new prompt, tool or knowledge source make anything worse?
- Acceptance: does the agent meet the agreed quality bar before it touches customers or staff?
- Operations: does quality hold when the provider rolls out a new model version?
If you are rolling out AI agents for quoting, service or sales, evals become non-negotiable the moment the agent acts on its own: creating records, sending emails or quoting prices.
How does an eval work? The building blocks
An eval combines tasks, trials, graders and a summary metric. Each task has a defined input and success criterion; the agent attempts it several times; graders check the result and the route taken; and a metric rolls it all up. Anthropic's guide to agent evals uses the same vocabulary.
Demystifying evals for AI agents, published in January 2026, breaks an eval into:
- Task: a single test with defined inputs and success criteria, such as "turn this purchase enquiry into a three-line quote".
- Trial: one attempt at a task. Because outputs vary, each task is attempted more than once.
- Grader: logic that scores one aspect of performance. A task can have several graders, each with several checks.
- Transcript (trace, trajectory): the full record of a trial, including outputs, tool calls and intermediate results.
- Outcome: the final state of the environment after the trial.
The outcome is what separates agent evals from chatbot evals. Anthropic's example is a booking agent: what matters is not that it says the flight is booked, but whether the reservation exists in the database. In a business setting, check that the ticket really sits in the right queue with the right status, not that the agent claims it does.
A single test case can live in a simple file:
id: quote-017
input: "Email of 3 Oct: 40 units of bracket type B, delivery week 46"
expected:
line_items: 1
sku: "HB-200"
quantity: 40
checks:
- type: code # compare fields against the record in the system
- type: model # rubric: tone, completeness, no invented discounts
trials: 5
Which kinds of graders are there?
There are three kinds: code-based checks, models acting as judges, and humans. Code is fast and unambiguous but rigid. Model judges capture nuance but must be calibrated. Human experts give the most reliable verdicts but are slow and costly, so their main job is to calibrate the other two.
Code-based graders compare fields, validate formats, run tests or inspect which tools were called in which order. The Claude documentation on evaluations describes them as the fastest and most reliable option, but lacking nuance for judgements that are not rule-based. Anthropic's agent guide adds that they are brittle when a valid answer is phrased differently from the expected pattern.
Model-based graders, often called LLM-as-a-judge, score answers against a rubric, compare two answers side by side or check natural-language assertions. The Claude documentation gives three tips: write detailed, clear rubrics; ask for a specific output such as "correct/incorrect" or a 1 to 5 score; and let the grader reason before it scores. OpenAI's evaluation best practices likewise favour pairwise or pass/fail judgements and advise controlling for response length, because judges tend to prefer longer answers.
These biases are well documented. The paper Judging LLM-as-a-Judge by Zheng et al. names position, verbosity and self-enhancement bias, while also reporting that strong model judges reached over 80% agreement with human preferences, roughly the level at which humans agree with each other.
Human grading by domain experts remains the reference point. In practice, experts grade a sample and you adjust the automated graders until their verdicts line up. Only then is it worth running them at scale.
| Grader | Strength | Weakness | Typical use |
|---|---|---|---|
| Code | fast, objective, reproducible | rigid with valid variations | fields, status, tool calls |
| Model | nuance, scales well | non-deterministic, needs calibration | tone, completeness, reasoning |
| Human | most reliable | slow, expensive | samples, calibration |
What changes when you evaluate agents rather than chatbots?
With a chatbot you mostly grade the answer. With an agent you also grade the route and the end state of the systems it touched: which tools it called, whether it followed policy, how many steps it took, and whether it reaches the same result consistently across repeated runs.
Three layers of measurement work well for agents:
- Outcome: is the end state right? The record in the ERP, the ticket status, the generated document.
- Route: did the agent use the right tools in a sensible order, avoid needless writes and respect approval rules? Anthropic's example for conversational agents is a constraint like finishing in fewer than 10 turns.
- Consistency: does the task succeed every time, not just once?
Two metrics cover the third layer. pass@k is the probability that at least one of k attempts succeeds, which is useful when a person picks from several drafts. pass^k is the probability that all k attempts succeed, which is what matters when the agent acts unsupervised. pass^k was introduced with the τ-bench benchmark, where even state-of-the-art function-calling agents of the time scored below 25% on pass^8 in the retail domain.
The gap is bigger than intuition suggests. As Anthropic's guide works out, a 75% per-trial success rate gives only about a 42% chance that three trials in a row succeed (source). An agent that is "usually right" is often not reliable enough for an unattended process.
Which types of eval suites do you need?
You need two: capability evals that show what the agent cannot yet do well, and regression evals that make sure it does not lose what it already does. Capability suites start with a low pass rate; regression suites should pass almost completely.
Anthropic frames it this way. Capability evals ask what the agent does well and should start at a low pass rate, giving the team something to climb. Regression evals ask whether the agent still handles everything it used to, and should sit at nearly 100%. Capability tasks that pass reliably graduate into the regression suite.
It also helps to split cases by content:
- Core cases: your most frequent real requests, such as standard quotes or recurring fault reports.
- Edge cases: incomplete, ambiguous, very long or irrelevant inputs. The Claude documentation explicitly lists irrelevant, overly long and ambiguous inputs among them.
- Negative cases: situations where the agent must not act: no discount promises, no deletions, hand over to a person. Anthropic warns that one-sided evals lead to one-sided optimisation, such as an agent that runs a search for almost every question.
What this overview leaves out on purpose is plumbing: how evals run in a CI pipeline and how they sit alongside unit and integration tests belongs to the development topic area. This page is about what to measure and when a result is good enough.
What do you need in place first?
You need real business cases as test material, domain experts who can say unambiguously what is correct, a sandbox where the agent can act without consequences, and complete logs of every run. Without these, your evals measure the test data more than the agent.
Data: the most telling cases come from production: ticket queues, shared inboxes, existing manual checks and bug reports. OpenAI calls out datasets that do not reflect production traffic as a common mistake, and the Claude documentation asks for evals that mirror your real-world task distribution. Pseudonymise personal data before using it.
Expertise: every case needs a clear definition of success. Anthropic proposes a simple quality check: two domain experts should independently reach the same pass/fail verdict. If they do not, the task is underspecified.
Environment: agents with write access need a copy or sandbox of the ERP, CRM or ticketing system that can be reset to a known state before each trial. Otherwise trials contaminate each other.
Logging: store every run with its input, tool calls, intermediate steps and outcome. OpenAI recommends logging everything during development so that you can mine the logs for new test cases.
Ownership: decide who sets thresholds, who signs off results and who decides when a regression appears. Acceptance needs both the business team and IT.
How do you build an eval programme step by step?
Start small: define success, collect 20 to 50 real cases, build simple graders, run each case several times, check results against expert judgement, and only then write down release thresholds. From there, every new production failure becomes a new test case.
- Define success. What does good enough mean for this process? For example: correct SKU and quantity, no invented terms, hand-off to a person for complaints. Each criterion must be checkable.
- Collect cases. Anthropic says 20 to 50 simple tasks drawn from real failures is a great start, and that teams often wait too long because they think they need hundreds. Mix core, edge and negative cases.
- Build graders. Check in code whatever code can check. Use a model rubric with a constrained output for the rest. The Claude documentation favours automated grading and prefers more cases with slightly noisier automated scores over a few hand-graded ones.
- Run multiple trials. Several attempts per case, so you see consistency (pass^k) rather than lucky hits.
- Read the transcripts. Inspect failures by hand. Anthropic notes that a 0% pass rate across many trials more often signals a broken task than an incapable agent. Graders have bugs too.
- Calibrate model judges. Have experts grade a sample and compare. Scale the judge only once agreement is stable.
- Set thresholds. For example: regression suite fully green, core cases at an agreed pass^k rate, zero violations on negative cases. The bar depends on what a failure costs you.
- Keep it running. Re-run on every change to prompt, model, tool or knowledge base, and turn each production failure into a new case.
In our own operations, writing thresholds down before the first run has proved its worth. Teams that set them after seeing results tend to drift towards whatever the agent currently achieves.
What are the limits and risks?
Evals only measure what is in the test set. A small, clean or stale set gives false confidence, model judges can be systematically wrong, and high pass rates may simply mean the tasks have become too easy. Evals also do not replace production monitoring, which the guide to AgentOps covers.
- Coverage: anything missing from the set goes unmeasured. New request types in production will not show up in old evals.
- Overfitting: optimise against the same set long enough and the team learns the cases, not the problem. Keep a held-out set that is used only for acceptance.
- Saturation: Anthropic describes how public benchmarks such as SWE-bench Verified approach saturation within a year; there, scores rose from about 40% to over 80%. Internal sets saturate too; that is the signal to add harder cases.
- Broken graders: a grader that rejects valid solutions makes a good agent look bad. Anthropic describes a case where a benchmark score jumped substantially once grading bugs were fixed.
- Biased model judges: position, verbosity and self-enhancement bias are documented. A code comment in the Claude documentation recommends grading with a different model from the one that produced the output.
- Cost: many cases times many trials times model judges adds up to a lot of API calls. Budget for it (see optimising AI agent cost and latency) and keep a fast core suite separate from the full acceptance suite.
When this approach does not fit: for an internal prototype tested by three people, a full eval suite is overkill; a dozen documented cases will do. And where domain experts cannot agree on what correct looks like, evals will not give you a clean number. They will first tell you that the success criterion needs work.
How do you get started?
Pick one process where the agent already runs or soon will, and collect 20 real cases with their correct outcomes this week. Decide what each case checks, run every case five times, and read the failures.
A pragmatic start in three moves:
- A spreadsheet is enough. Columns: input, expected outcome, check type, result per trial. Tooling can come later.
- Failures are raw material. Every complaint and every manual correction from production becomes a test case.
- One number for release. Agree with the business owner which rate must be reached before the agent may act without human approval.
If you are planning agents for specific workflows such as quotation or technical service, collect eval cases while you map the process. The question "what exactly is a correct result here?" often clarifies the process more than any requirements document.
Frequently asked questions
How many test cases do I need to start?
Anthropic suggests 20 to 50 simple tasks drawn from real failures as a good start. Coverage of your real traffic matters more than volume.
Is it safe to use an LLM as a judge?
It is common practice for open-ended outputs, but only after you have checked its verdicts against human experts. LLM judges have documented biases, for example towards longer answers.
What is the difference between pass@k and pass^k?
pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k attempts succeed, so it measures consistency rather than capability.
Do evals replace unit and integration tests?
No. Conventional tests cover the deterministic code around the agent; evals cover the model's non-deterministic behaviour. Production agents need both.
When should evals run?
On every change to prompts, models, tools or the knowledge base, and whenever your provider ships a new model version you plan to adopt.
Sources
- Anthropic Engineering: Demystifying evals for AI agents (9 January 2026) anthropic.com
- Claude Docs: Define success criteria and build evaluations platform.claude.com
- OpenAI API Docs: Evaluation best practices developers.openai.com
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) arxiv.org
- Yao et al.: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) arxiv.org