Testing AI Agents: From Unit Tests to Eval Suites
Testing an AI agent is awkward because half of it is ordinary software and the other half is a language model that answers slightly differently every time. Treat both halves the same way and you end up with flaky tests that everyone ignores, or with tests so loose they prove nothing. This guide lays out a test pyramid for agents and shows how to run it as a normal engineering practice rather than a one-off quality check.
What does testing an AI agent actually involve?
Testing an agent means answering two separate questions. Does the code around the model work correctly? And, given that code, does the model reliably make the right decisions? Classic tests with a single right answer handle the first. Evals, which deal in success rates rather than pass or fail, handle the second.
Take a support agent that reads an incoming ticket, looks up the customer's contract in a CRM, checks the warranty terms and drafts a reply. Parsing the ticket, calling the CRM API, computing the warranty end date and saving the draft are plain code. Deciding which contract applies when the customer writes "the machine we bought last spring" is a model decision.
Agents also keep changing after launch. Prompts are refined, model versions are swapped, tool descriptions are edited and data sources drift. Any of these can quietly break behaviour that worked last week, which is why testing belongs in the day-to-day development loop, just as it does for any other production system.
How this differs from related material: Evals for AI Agents: Making Quality Measurable covers how to define and measure quality in the first place – metrics, graders and datasets. This article is about the wider test strategy: how evals fit alongside unit, integration and regression tests so the whole thing runs inside your development process.
What does a test pyramid for agents look like?
The agent test pyramid has four layers. At the bottom sit many fast, deterministic unit tests. Above them come component tests of individual model calls, then integration tests of full agent runs against test systems, and at the top a regression suite built from real cases. Higher layers are slower, costlier and statistical; lower layers are numerous and exact.
Layer 1: deterministic unit tests
Everything that is not a model call gets tested the usual way: tool functions, parsers for model output, the function that assembles the prompt from a template and context, tool argument validation and permission checks. These tests run in milliseconds, cost nothing and are fully reproducible.
At this layer you replace the model with a stub that returns prepared responses. In Python, the pytest monkeypatch fixture exists precisely for code that depends on network access or global settings.
# test_tools.py – the model is stubbed, never called
from agent.runner import run_step
def test_unknown_tool_is_rejected(monkeypatch):
fake_reply = {"type": "tool_use", "name": "delete_customer", "input": {}}
monkeypatch.setattr("agent.runner.call_model", lambda *a, **k: fake_reply)
result = run_step("Close ticket 4711")
assert result.status == "rejected"
assert "delete_customer" in result.reason
def test_valid_lookup_passes_arguments(monkeypatch):
fake_reply = {"type": "tool_use", "name": "get_contract", "input": {"customer_id": "C-88"}}
monkeypatch.setattr("agent.runner.call_model", lambda *a, **k: fake_reply)
result = run_step("Which contract does C-88 have?")
assert result.tool_args == {"customer_id": "C-88"}
Failure paths are where these tests earn their keep. What happens when the model calls a tool that does not exist, returns a field in the wrong format or stops halfway? All of these happen in production, and your code has to handle them cleanly.
Layer 2: component tests of single model calls
The second layer calls the real model, but only for one narrow step: a classification, an extraction, a tool choice. Assertions target properties of the response rather than exact wording – "is valid JSON", "contains customer ID C-88", "picks the get_contract tool".
Tools such as promptfoo ship these checks ready-made. Its assertion reference lists deterministic checks such as contains, regex, is-json, is-valid-function-call and custom python or javascript functions, budget checks such as latency and cost, and model-assisted checks such as llm-rubric and factuality. For agents there are trajectory checks as well, including trajectory:tool-used and trajectory:tool-sequence.
Layer 3: integration tests of full runs
An integration test lets the agent complete a whole task against test systems: a staging CRM, a test mailbox, a sandbox ticketing instance. At the end you inspect the state of those systems. Is there a draft reply with the right contract reference? Was nothing deleted? Was no email actually sent when only a draft was allowed?
The research benchmark τ-bench is built on the same idea: it compares the database state at the end of a conversation with an annotated goal state instead of scoring the dialogue line by line.
Layer 4: regression suite from real cases
The top layer is a collection of real tasks the agent has handled correctly, or once failed and was then fixed. This suite runs on every meaningful change. Its job is not to measure new abilities but to make sure solved problems stay solved.
How do you handle non-deterministic outputs?
You handle non-determinism by running each case several times and judging a success rate rather than a single result, and by writing checks that accept every valid variant. Check properties and outcomes, not exact text. A flaky test then turns into a measured reliability figure you can track over time.
Repeat, and pick the right metric
Two metrics have become standard. According to Anthropic's engineering guide to agent evals, pass@k estimates the chance that at least one of k attempts succeeds, while pass^k estimates the chance that all k attempts succeed. If a human reviews every output anyway, pass@k may be enough. If the agent answers customers or writes data on its own, pass^k is the number that matters.
The gap is larger than intuition suggests. Using the worked example from Anthropic, a 75% per-trial success rate gives only about 42% for pass^3. The τ-bench authors reported that function-calling agents that were state of the art at the time reached a pass^8 below 25% in their retail domain. An agent that works "most of the time" is often far less dependable than one green test run implies.
In promptfoo, the --repeat option runs each test several times, as described in the command-line reference. Make sure a response cache isn't quietly serving the same answer each time; --no-cache disables it.
Grade the outcome, not the route
Agents regularly find different, equally valid ways to reach a goal. Anthropic advises that it is often better to grade what the agent produced than the path it took. A test that insists on a fixed sequence of tool calls will break every time someone makes a harmless improvement.
Check the route only where the route is itself a requirement. An approval tool must run before a payment, a delete tool must never appear in a given workflow, personal data must not be passed to an external service. Rules like these can be written as hard assertions and belong in every suite. In production, guardrails for AI agents enforce the same rules; your tests confirm they actually fire.
Choose graders deliberately
There are three kinds of grader. In Anthropic's breakdown, code-based graders are fast, cheap and reproducible but brittle when a valid answer looks different; model-based graders are flexible but non-deterministic themselves and need calibrating against human judgement; human grading is the gold standard but slow and expensive.
A workable rule: check as much as possible in code, use a model grader only for what can't be reduced to rules (tone, completeness of a summary), and use human spot checks to keep the model graders honest.
Which tests matter most for which kind of agent?
The weighting depends on what the agent does. A read-only research agent mostly needs evals for answer quality and faithfulness to its sources. An agent that writes into business systems mostly needs integration tests on the final system state and hard checks on which actions it may take.
- Knowledge and research agents (questions over manuals, policies, project files): focus on retrieval accuracy and on whether answers stay faithful to the documents found. Unit tests cover document chunking and indexing. More on this kind of agent: AI agents for knowledge management.
- Process agents with write access (quotations, tickets, purchase orders): focus on integration tests against sandboxes, permission checks and pass^k for the core workflows. See AI agents for quotation for typical flows.
- Conversational agents facing customers (service, sales): add simulated users, as τ-bench does, plus checks on tone and policy adherence, usually handled by a model grader.
- Multi-step agents with sub-agents: add tests at the hand-off points – whether a sub-agent receives what it needs and whether its answer is processed correctly afterwards.
For an overview of where agents are used, see AI agents.
What do you need in place before you start?
A workable agent test strategy needs three things: an architecture that separates model calls from the rest of the code, test environments for every system the agent writes to, and a set of real inputs with known correct outcomes. Without them, testing collapses into manual trial and error.
Architecture
Put the model call behind a thin interface that tests can swap out. Keep tools as plain functions with clear inputs and outputs, not methods that quietly change global state. Make every agent run produce a full trace: input, prompts, tool calls with arguments, tool results and final output. Those traces become the raw material for new test cases.
Test environments
Integration tests need systems where the agent can't damage anything real: a staging CRM, a test mailbox, a copy of the document store with pseudonymised data. Where no sandbox exists, a fake service that mimics the real API for the cases under test will do – with the risk that it drifts away from the original.
Test data
The most valuable cases come from production: real requests the agent got wrong, and real requests that represent everyday traffic. Anthropic recommends starting with 20 to 50 simple tasks drawn from real failures instead of waiting until you have hundreds. Every case needs an unambiguous definition of success. Pseudonymise personal data in test cases so the suite can be operated in line with the GDPR and comparable privacy laws.
People
Someone from the business side has to define what a correct result looks like. An engineer cannot judge whether a warranty decision is right or whether a quotation uses the right discount tiers. Budget that time explicitly, or the eval suite has nothing solid to stand on.
How do you roll this out step by step?
Roll it out in five steps: unit tests for deterministic code first, then a small eval suite from real cases, then repeated runs with a success threshold, then CI integration, and finally a routine that turns every production failure into a new test case. Each step is useful on its own, so you can stop after any of them.
- Isolate and test the deterministic code. Tools, parsers, prompt assembly and validation get unit tests with a stubbed model. These run on every commit.
- Build a first eval suite. 20 to 50 real cases, each with an input and a success criterion. For each case decide whether it is graded by code, by a model grader or by a human spot check.
- Repeat and set a threshold. Run every case several times and decide whether pass@k or pass^k applies. For workflows with write access, set a minimum rate below which nothing ships.
- Wire it into CI. The fast regression suite runs on every change to prompts, model version, tool schema or retrieval settings. Larger suites with many repetitions run nightly or before a release to keep time and API spend under control.
- Feed production failures back. Every wrong answer in production, together with its trace, becomes a test case. The suite then grows along the agent's real weak spots.
Keep two kinds of eval apart. Anthropic distinguishes capability evals, which should start at a low pass rate, from regression evals, which should pass almost every time. Capability evals show what you are working towards; regression evals protect what already works. Once a capability eval passes reliably, move it into the regression suite.
One lesson from our own operations: before tuning any grader, read the traces of failed runs yourself. Often the agent is fine and the grader is rejecting a valid answer. Anthropic makes the same point – you only know whether your graders work if you read the transcripts and grades from many trials.
What are the limits and risks?
Tests lower risk; they do not remove it. An eval suite only measures the cases it contains, so rare inputs, new request types and slowly drifting data stay invisible. Add the cost of repeated model calls and the possibility that graders themselves get it wrong, and it is clear that testing needs to be paired with monitoring.
- Coverage: a 50-case suite says little about case 51. Pair tests with production monitoring: sampling, user feedback, alerts on unusual tool calls. AgentOps: Running AI Agents Reliably covers that side.
- Grader errors: model graders can be too lenient or too strict. Re-check them against human ratings regularly.
- Overfitting: tweaking prompts until exactly the test cases pass improves the suite, not the agent. Keep a held-out set that is never used for tuning.
- Cost and run time: many repetitions with large models cost money and time. Separate quick per-commit suites from large release suites.
- Stale test systems: fakes and sandboxes drift from the real system. A green integration test against an outdated stub is misleading.
- Model upgrades: a new model can score better overall and still lose individual cases that used to be safe. That is exactly what the regression suite is for, so run it in full before every model switch.
When the full pyramid isn't worth it: for a prototype that only drafts suggestions for humans and will be thrown away in a few weeks, unit tests for the tools and a handful of manual checks are enough. The full structure pays off once the agent runs permanently or writes into real systems.
Where do you start?
Start with the agent that runs most often or writes into your most important systems. Collect a week of real inputs plus every case where a user had to correct the result. That gives you the first unit tests for failure paths and the first twenty eval cases.
A pragmatic first day:
- Turn on full tracing for agent runs if it isn't already on.
- Cover the three most important tools with unit tests, including invalid arguments.
- Write down ten real cases with clear success criteria and run each one five times.
- Record the success rate per case and investigate the unstable ones first.
From there the suite grows with every production failure. For a deeper look at defining quality criteria and graders, read Evals for AI Agents. If you are still deciding which agent is worth building in the first place, the AI agents overview covers the typical use cases.
Sources and tools referenced as of November 2026.
Frequently asked questions
Can I make agent tests deterministic by setting temperature to zero?
Don't rely on it. A new model version, an edited prompt or different context will change outputs anyway, so design tests that tolerate variation rather than trying to remove it.
How large should a first eval suite be?
Anthropic suggests starting with 20 to 50 simple tasks drawn from real failures. Realistic inputs and unambiguous success criteria matter more than volume.
Should evals run in CI?
The fast regression suite should, on every change to prompts, model versions or tool schemas. Large capability evals with many repetitions are better scheduled nightly or before a release because of time and API spend.
Should I test the agent's path or only its result?
Mostly the result, such as the final state of the target system. Check the path only where the path itself is a rule, for example an approval step that must happen before a booking.
Sources
- Anthropic Engineering: Demystifying evals for AI agents (January 2026) anthropic.com
- Yao et al.: τ-bench – A Benchmark for Tool-Agent-User Interaction in Real-World Domains arxiv.org
- promptfoo: Assertions and metrics promptfoo.dev
- promptfoo: Command line promptfoo.dev
- pytest: How to monkeypatch/mock modules and environments docs.pytest.org