# Optimizing Cost and Latency of AI Agents

> Measure tokens and response time per step first. Then five levers apply: prompt caching, a model cascade, batch processing for work that can wait, a lean context and, at high volume, self-hosted inference. Test every change against a fixed evaluation set so quality does not quietly drop.

Canonical URL: https://www.ruebstudio.de/en/guides/ai-agent-architecture/ai-agent-cost-latency-optimization  
Author: Dr.-Ing. Marcus Rüb, CEO  
Published: 2026-10-28

A chatbot calls a language model once per question. An agent may call it ten or twenty times for a single task: plan, call a tool, read the result, decide the next move. Each call resends the whole conversation so far. That is why agent bills and response times grow much faster than those of a simple assistant. This guide covers the five main levers, how they interact, and where each one can hurt quality. Vendor details are as of October 2026.

## What does cost and latency optimisation mean for an agent?

It means finishing the same task with fewer billed tokens and less waiting, without worse results. Cost comes almost entirely from input and output tokens per model call. Latency comes from processing the input, generating the output, and the number of steps the agent runs one after another.

Take a support agent that triages an incoming ticket, looks up the customer's contract in the CRM, checks recent orders, and drafts a reply:

1. The agent receives its system prompt, the tool definitions and the ticket.
2. It calls a tool, for example a CRM lookup.
3. The tool result is appended to the context, and the next call sends everything again.
4. After several rounds it writes a summary and a draft reply.

The system prompt and tool schemas travel with every single call. With a long tool list, they quickly become the largest part of the input. That is where most levers apply. In practice, optimisation is rarely about "a cheaper model" and mostly about "less repeated work".

## Where do an agent's cost and delay come from?

Cost is input plus output tokens, multiplied by the model's price, summed across every step. Delay is the time to first token, plus the time to generate the output, plus the tool calls in between. If you do not measure per step, you are optimising blind.

Log three things for every model call:

- **Input tokens**, split into freshly processed and read from cache. Major providers return these in the response's `usage` object.
- **Output tokens**, including any reasoning tokens the model produces.
- **Timings**: time to first token, total call duration, and the duration of the tool call that follows.

You can count input before sending it. Anthropic offers a dedicated endpoint that, according to its [token counting documentation](https://platform.claude.com/docs/en/build-with-claude/token-counting), is free to use, covers system prompts, tools, images and PDFs, and returns an estimate. One trap when switching models: according to [the same page](https://platform.claude.com/docs/en/build-with-claude/token-counting), newer Claude models use a different tokenizer, and the same text produces roughly 30 percent more tokens than on earlier models. Old measurements do not carry over.

With this data you see which step is expensive and which is slow. They are often different steps: long context drives cost, long output drives waiting time.

## What levers reduce cost and latency?

There are five levers with different effects: prompt caching, model cascades, batch processing, context hygiene and self-hosted inference. Caching and context hygiene help cost and latency at once. Cascades mainly save money. Batching saves money but adds waiting. Self-hosting turns variable cost into fixed cost.

### Prompt caching

With caching, the provider keeps the processed state of an unchanged prompt prefix and reuses it on the next request. For Claude, the [Anthropic prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) states that the prefix is built in the order `tools`, `system`, `messages`. Writing to the cache costs 1.25 times the base input price for a five-minute lifetime and 2 times for one hour; a cache read costs 0.1 times by default, and less on some current models. The same page states that the response itself is unaffected.

OpenAI turns caching on automatically for supported models. Its [prompt caching guide](https://developers.openai.com/api/docs/guides/prompt-caching) says caching starts at 1,024 tokens on current models and that cached input tokens are discounted by up to 95 percent. Both vendors give the same advice: put stable content first and changing content, such as timestamps or user details, last.

For agents, that means: do not rebuild tool definitions or the system prompt per call, keep timestamps out of the system prompt, and keep tool order fixed.

### Model cascades and routing

A cascade sends each request to a small, cheap model first and escalates to a larger model only when the answer looks uncertain or inadequate. Routing is a variant: a classifier decides upfront which model gets the task. The research paper [FrugalGPT by Chen, Zaharia and Zou](https://arxiv.org/abs/2305.05176) reported that a cascade could match the strongest single model in their experiments with up to 98 percent lower cost. That is a lab result on specific datasets, not a promise for your workflow.

A split by step type tends to work well: classification, field extraction and formatting on a small model; planning and judgement on a large one. Choosing the right model per task is the subject of a separate overview on model selection, coming soon.

### Batch processing

Not every task needs an answer in seconds. Overnight reports, pre-sorting a shared inbox, or enriching master data can wait. Anthropic's [Message Batches API documentation](https://platform.claude.com/docs/en/build-with-claude/batch-processing) lists a 50 percent discount, says most batches finish in under an hour, and expires requests not completed within 24 hours. OpenAI's [Batch API guide](https://developers.openai.com/api/docs/guides/batch) also lists a 50 percent discount and a 24-hour window, with separate rate limits.

Batching only fits agent work where a step needs no feedback loop. A multi-step agent with tool calls cannot run as one batch, but independent subtasks can.

### Context hygiene

Every round lengthens the context. Tool results, such as a full JSON payload from your ERP or CRM, stay in the history and are paid for again on every later call. Context hygiene means trimming tool responses to the fields the agent needs, summarising finished sub-steps, and retrieving relevant passages instead of pasting whole documents, as covered in the guide to [enterprise RAG architecture](https://www.ruebstudio.de/en/guides/ai-agent-architecture/enterprise-rag-architecture).

This is not only about money. The study [Lost in the Middle by Liu et al.](https://arxiv.org/abs/2307.03172) found that language models often use information at the start or end of a long context better than information in the middle. A shorter, cleaner context is frequently a better one.

### Self-hosted inference

If you run an open-weight model yourself, you pay for hardware, power and operations instead of tokens. Serving engines such as vLLM use their own techniques: the [PagedAttention paper by Kwon et al.](https://arxiv.org/abs/2309.06180) reported 2 to 4 times higher throughput than the comparison systems of the time at the same latency. vLLM can also reuse already processed prompt prefixes, as described in its [prefix caching documentation](https://docs.vllm.ai/en/latest/design/prefix_caching.html), which is the same idea as provider-side caching.

Self-hosting mainly makes sense at high, steady volume or when data must stay on your own infrastructure. For sizing, see the guide to [hardware for local LLMs](https://www.ruebstudio.de/en/guides/local-ai/hardware-local-llm-gpu-server-sizing); for background on running models in your own environment, see [sovereign AI](https://www.ruebstudio.de/en/sovereign-ai).

## How do the levers interact, and where does quality suffer?

The levers do not simply add up; they affect each other. Trimming context can invalidate the cache if it changes the stable prefix. A cascade splits caching across models, because each model keeps its own cache. And any lever can cost quality if you introduce it without measuring.

The key interactions:

| Combination | What happens | What to watch |
|---|---|---|
| Caching + context hygiene | Trimming in the middle of the prompt invalidates the cache from that point | Trim only at the end, or establish a new stable prefix afterwards |
| Caching + cascade | Each model builds its own cache | Limit the cascade to steps with short context |
| Batch + caching | Short-lived cache entries can expire between batched requests | Anthropic's documentation states cache pre-warming is not supported inside a batch |
| Cascade + quality | The small model gives fluent but wrong answers | Check the escalation rule, not just the model's self-assessment |
| Self-hosting + quality | Open-weight models can be weaker on some tasks | Use the same evaluation set as for the hosted model |

Protect quality with a fixed evaluation set, often called [evals](https://www.ruebstudio.de/en/glossary/ai-evals): 50 to 200 real, anonymised cases from the workflow with a known correct outcome. Every change to model, prompt or context runs against this set first. If the error rate rises, drop the change, however much it saves.

## What do you need before you start optimising?

You need three things: logging per model call, an evaluation set with known correct answers, and an architecture where model, prompt and context assembly can be swapped. Without any one of these, optimisation turns into guesswork, and quality problems surface only in production.

**Data.** Log per call: agent, step, model, input tokens (fresh and cached), output tokens, time to first token, total time, and the quality check result. A simple database table is enough to start; the guide to [AgentOps](https://www.ruebstudio.de/en/guides/ai-agent-development/agentops-running-ai-agents) shows how this grows into proper operations.

**Systems.** The agent should pick its model per step from configuration, not hard-coded. Tool responses should be trimmed in one layer you can change centrally. For batch work, you need a way to collect results asynchronously and feed them into the next stage.

**Ownership.** Someone has to define "good enough". A customer quotation needs a different standard from an internal ticket pre-sort. Without that standard, there is no basis for trading cost against quality. For concrete workflows, see the examples on [agents for technical service](https://www.ruebstudio.de/en/ai-agents/technical-service) and [quotation agents](https://www.ruebstudio.de/en/ai-agents/quotation).

## What is a sensible step-by-step approach?

Work in a fixed order: measure first, then apply the levers that carry no quality risk, then the ones that do. Introduce each change on its own and check it against the evaluation set. That way you know where every effect comes from and can roll it back if needed.

1. **Measure the baseline.** Log one week of production. Output: cost and latency per task and per step, and the three most expensive steps.
2. **Order the prompt for caching.** Stable tool definitions and system prompt first, changing content last. Check the cache hit rate in the `usage` object.
3. **Trim tool responses.** Return only the fields the agent needs for its next step. Run the evaluation set.
4. **Constrain output.** Use structured output with a fixed schema instead of free text where it fits. Output is generated token by token, so long answers add directly to waiting time.
5. **Merge or parallelise steps.** Run independent tool calls at the same time rather than in sequence. This shortens total time without adding cost.
6. **Move non-urgent work to batch.** Overnight runs, bulk analysis.
7. **Introduce a cascade.** A smaller model for simple steps, with a clear escalation rule. Run the evaluation set and look at each failure.
8. **Assess self-hosting.** Only once volume and data constraints justify it and steps 1 to 7 are done.

In our own operations, it has paid off to let each change run in production for a few days before adding the next. Some effects only show up in edge cases the evaluation set does not cover.

## What are the limits and risks?

The biggest risks are unnoticed quality loss, dependence on one provider's pricing model, and optimising the wrong thing. Looking only at token cost hides the fact that one wrong output usually costs more than the tokens you saved. Prices and cache rules also change with each new model generation.

- **Silent quality loss.** A smaller model produces fluent text that is wrong. Without an evaluation set, a customer finds it first.
- **Outdated cache assumptions.** Minimum lengths, lifetimes and discounts differ by model and provider. Anthropic's documentation, for instance, lists minimum cacheable lengths from 512 to 4,096 tokens depending on the model. Recheck the hit rate after every model change.
- **Batch and deadlines.** If a batch does not finish in time, requests expire. Batching is unsuitable for workflows with fixed commitments.
- **Underestimating self-hosting.** Hardware, updates, monitoring, on-call duty and utilisation all cost time and money. With uneven demand, expensive GPUs often sit idle.
- **Wrong focus.** If an agent takes twenty steps for a job that needs five, the design is the problem, not the token price.
- **Logging and privacy.** Logs containing prompt content can include personal data. Store metrics only, or pseudonymise content, so the setup can be operated in line with the GDPR and similar laws.

## How do you get started?

Start with one agent that already runs in production and measure cost and waiting time per step for a week. Then reorder the prompt for caching and trim tool responses. Neither carries quality risk, and together they quickly show how much potential there is.

In parallel, build the evaluation set. Every further lever depends on it, and it pays for itself regardless of cost, because it protects every future change to the agent.

Only then move on to cascades, batching and, where justified, self-hosting. In-depth articles on each topic, including forecasting token costs and cost control across a fleet of agents, will appear over time in the [guides section](https://www.ruebstudio.de/en/guides). For an overview of what agents can do and how they are built, see [AI agents](https://www.ruebstudio.de/en/ai-agents).

## Frequently asked questions

### Which lever usually pays off first for an agent?

Prompt caching, if the system prompt and tool definitions are long and stable. It does not change the model's answers and needs no rework of the agent logic.

### Will quality suffer if I switch to a smaller model?

It depends on the step. Small models often handle classification, extraction and reformatting well, and multi-step planning less well. Decide with an evaluation set, not by instinct.

### When does self-hosted inference make sense?

When volume is high and steady, or when data must not leave your infrastructure. Compare hardware, power, operations and utilisation honestly with what you pay per token.

### How should I measure an agent's latency?

Track time to first token, total time per model call and total time per task across all steps separately. An agent that takes ten steps feels slow even if each call is fast.

## Sources

- [Anthropic: Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
- [Anthropic: Batch processing (Message Batches API)](https://platform.claude.com/docs/en/build-with-claude/batch-processing)
- [Anthropic: Token counting](https://platform.claude.com/docs/en/build-with-claude/token-counting)
- [OpenAI: Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
- [OpenAI: Batch API](https://developers.openai.com/api/docs/guides/batch)
- [Chen, Zaharia, Zou (2023): FrugalGPT](https://arxiv.org/abs/2305.05176)
- [Kwon et al. (2023): Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
- [vLLM: Automatic Prefix Caching](https://docs.vllm.ai/en/latest/design/prefix_caching.html)
- [Liu et al. (2023): Lost in the Middle](https://arxiv.org/abs/2307.03172)
