Skip to content

AgentOps: Running AI Agents Reliably

Go-live is where an AI agent starts doing real work, not where the project ends. From that day on, models, APIs, data and the business process itself keep changing, even if nobody touches the agent's code. AgentOps is the practice of running agents so they still do the job they were built for a year later.

What is AgentOps and why does it matter?

AgentOps is the operational discipline for AI agents: ownership, observability, maintenance, change management and incident handling for software that uses a language model to plan and execute several steps, calling tools along the way. It takes established IT operations practice and adapts it to systems whose behaviour is not fully deterministic.

Three properties make agents harder to run than conventional applications:

  • Non-deterministic output. The same input can lead to different answers and different tool calls. A green health check tells you the service is up, not that its results are right.
  • Multi-step execution. A typical agent reads a request, searches internal documents, queries an ERP system and drafts a reply. In its guide Building effective agents, Anthropic notes that the autonomy of agents means higher costs and the potential for compounding errors.
  • Dependence on someone else's model. In most setups you do not own the model. The provider improves it, replaces it and eventually switches it off.

AgentOps matters as soon as an agent moves beyond a pilot and takes over part of a live process: drafting quotations, triaging service tickets, answering questions from internal documentation or preparing sales outreach. For an overview of agent types used inside companies, see AI agents for business teams.

How does AgentOps work? The building blocks

AgentOps rests on six building blocks: ownership, observability, quality measurement, change management, incident management and cost control. None of them is new to IT operations, but each one needs adjusting for agents. When one is missing, the gap usually surfaces weeks after launch rather than on day one.

Ownership

Every agent needs two owners. The business owner is accountable for whether results are correct for the process. The technical owner is accountable for uptime, security and updates. Name both people before go-live. Without a business owner, nobody notices quality slowly slipping; without a technical owner, an expired API key can go unnoticed for days.

Observability

For every run you should be able to see the input, each model call and tool call, token usage, latency per step and the final output. Distributed tracing is the natural fit. The OpenTelemetry project maintains dedicated semantic conventions for generative AI, covering spans, metrics and events for GenAI clients as well as the Model Context Protocol (MCP). Instrumenting against an open convention keeps you free to switch observability tools later.

Quality measurement

Infrastructure metrics show that the agent is running, not that it is doing good work. Quality needs its own signals: a fixed set of real test cases with expected results (usually called evals), human review of random samples, and everyday signals such as how often staff discard or heavily rewrite an agent's draft.

Change management

Any change to a prompt, model, tool definition or knowledge source is a production change. Version it, test it against your evals before release, and roll it out in a way that lets you return to the previous state quickly if results get worse.

Incident management

Agents fail in two ways. Hard failures are visible: the model endpoint is unreachable or a tool returns an error. Soft failures are not: the agent runs, but its output is wrong or incomplete. Both need a defined response, covered in the section on limits and risks below.

Cost control

With hosted models you pay per token. An agent stuck in a loop, or one that loads far too much context on every run, produces costs nobody sees until the invoice arrives. The OWASP Top 10 for LLM Applications 2025 lists this as a risk in its own right, "Unbounded Consumption". Per-run and per-day budgets with alerts on outliers belong in every operating setup.

Where AgentOps applies: operating models

How much operational effort an agent needs depends less on technology than on how it is used: whether it only drafts or also acts, whether it faces staff or customers, and whether the model runs in the cloud or on your own infrastructure. Those three factors set the scope of AgentOps.

Drafting agents with human approval

The agent prepares, a person checks and approves, a pattern known as human-in-the-loop. Quotation drafts and suggested replies in customer service are typical. This is the easiest model to operate, because every error can be caught before it has an effect. The key operating metric is the rejection rate: when it rises, something has changed in the data, the prompt or the model. See the agents for quotations and technical service for concrete examples.

Acting agents with write access

Through tool use, the agent creates records, sends messages or changes statuses in other systems on its own. OWASP calls the underlying risk "Excessive Agency" and traces it to excessive functionality, excessive permissions or excessive autonomy over high-impact actions. Operationally, that means least-privilege credentials, a log entry for every write action, hard limits per run and a fast way to stop the agent.

Knowledge agents

Agents that answer questions from internal documents age with their sources. If new manuals are missing from the index, or outdated versions are still in it, answers stay fluent but become wrong. Operations here is mostly content work: curating sources, re-indexing and controlling which documents are searchable at all. The page on knowledge management agents shows how such agents are set up.

Hosted model or self-hosted

With a hosted API, the provider runs the model infrastructure but also decides when models are replaced. If you run open-weight models on your own servers, you choose when to upgrade, but you take on hardware, patching, security updates and capacity planning. The guide on sizing a GPU server for local LLMs covers the hardware side. The trade-offs around data control and infrastructure are covered on the sovereign AI page.

What do you need in place before go-live?

Four things should exist before an agent goes live: named owners, a set of real test cases, a trace of every run, and a documented fallback to manual handling. Without them you cannot tell whether the agent is doing well, and you cannot respond cleanly when it is not.

Data. Quality measurement needs real examples from the process, each paired with the result an experienced employee would have produced. A few dozen carefully chosen cases are a workable starting point; edge cases and difficult cases matter more than volume.

Systems. Give the agent its own technical identities and credentials rather than a shared user account. That is the only way to attribute write actions, limit permissions and rotate keys. You also need a place for logs and traces that the operations team can access.

Organisation. Decide who may approve prompt changes, who gets notified during an incident and who has the authority to switch the agent off. A short runbook is enough, as long as it answers those questions; it fits naturally into a lean AI governance framework.

Regulation. If an agent falls into a high-risk category under the EU AI Act, Article 26 requires deployers, among other things, to assign human oversight to competent people, monitor operation according to the provider's instructions for use, and keep automatically generated logs under their control for at least six months. Most back-office agents are not high-risk, but check your specific case with legal counsel; this section is orientation, not legal advice. Outside the EU, frameworks such as the NIST AI Risk Management Framework offer a voluntary structure for the same questions.

From go-live to steady state: a step-by-step approach

Moving an agent into steady-state operation works best in five steps: define operating targets, set up measurement, run a supervised ramp-up, agree a maintenance rhythm and plan model changes in advance. This order ensures you can measure before you start changing things.

1. Define operating targets

Write down what "working" means in numbers you can measure: share of runs without technical errors, share of drafts accepted without major edits, maximum run time, maximum cost per run. Google's SRE book describes the error budget: the target defines how much unreliability is acceptable over a period, and as long as budget remains, changes can ship. The same idea transfers well to quality errors in agents.

2. Set up measurement

Trace every run, build a dashboard around your targets and alert on hard failures and cost spikes. Run your evals automatically on every change and on a fixed schedule, for example weekly. The schedule matters because behaviour can shift without any change on your side, for instance when data in connected systems changes.

3. Run a supervised ramp-up

In the first weeks, domain experts review more output than they will later. Each deviation is categorised as a data issue, prompt issue, tool error, model limitation or process question. That list drives the first fixes and becomes a source of new test cases. Only once the error rate is stable within target does review drop to sampling.

4. Agree a maintenance rhythm

Maintenance is ongoing work for agents, not an exception. Recurring tasks include reviewing quality metrics, analysing rejected output, curating knowledge sources, updating tools after changes in connected systems, rotating credentials and comparing spend against budget. Assign each task an owner and a frequency.

5. Plan model and API changes ahead

Providers retire models on a regular basis. Anthropic's model deprecations page defines four lifecycle stages (Active, Legacy, Deprecated, Retired) and commits to at least 60 days' notice before retirement for publicly released models. An example from that page as of October 2026: deprecation of claude-sonnet-4-5-20250929 was announced on 30 September 2026, with retirement scheduled for 30 November 2026. Requests to retired models fail.

Models are not the only moving part. According to the same page, API version 2026-10-21 removed the temperature, top_p and top_k parameters, and requests that set them return an error on that version. An agent using those parameters has to be updated before it moves to the new API version. Plan for this by:

  • pinning model versions explicitly instead of relying on a "latest" alias,
  • tracking deprecation notices from every provider you use and putting the dates in your operations calendar,
  • treating each model switch like a release: run your evals against the new model, review the differences, adjust prompts, then cut over.

Limits and risks in production

The biggest operational risks with agents are gradual quality loss, unnoticed wrong actions in other systems, manipulation through input and creeping costs. They rarely come from one dramatic failure; they build up from many small changes that nobody looks at together. AgentOps exists to close exactly that gap.

Drift. The agent does not get worse because its code changes but because its environment does: new product names, revised forms, an ERP upgrade, a new model version. Without regular evals against a fixed set of cases, drift shows up as user complaints rather than as a metric.

Prompt injection. Prompt injection is first on the OWASP list for 2025. An agent that reads emails, web pages or uploaded files can encounter hidden instructions and act on them. In production you contain the damage with narrow permissions, approval steps before sensitive actions and logs detailed enough to reconstruct an incident.

Silent failures. A tool returns an empty result instead of an error, and the agent still writes a plausible answer, often an AI hallucination. You only catch these cases if tool results are visible in the trace and your checks flag suspiciously empty responses.

Provider dependence. A new model may perform better or worse on your specific task than the old one. Without evals you cannot tell before switching and have to rely on the provider's announcement.

When this is overkill. For an agent that produces a low-stakes summary a few times a month, a full AgentOps process is too much. A pinned model version, basic logging and a named contact person will do. Conversely, if a process tolerates no errors and cannot be protected by human approval, an agent may simply be the wrong tool.

What happens in an emergency. Every agent needs a switch that stops it immediately and a documented fallback: work returns to manual handling until the cause is fixed. Rehearse that fallback once before you need it.

How to get started with AgentOps

Start with one agent that is already live or close to launch, and sort out ownership, tracing and test cases first. Those three cover most of the risk and can usually be put in place within days. Dashboards and tooling can follow step by step afterwards.

A practical starting sequence:

  1. Inventory. List every agent and LLM-based automation in the company with its model and model version, connected systems, permissions and owners. For many organisations this is the first complete overview.
  2. Five metrics. Error rate, rejection rate, run time, cost per run and run volume. That is enough to begin with.
  3. Collect test cases. Ask the business team for around 30 real, anonymised cases with expected results. From then on, every change can be checked against them.
  4. Maintain a calendar. Record known retirement dates for the models and API versions you use, and schedule migrations with time to spare.

In our own operations, it has worked well to review operations and development in the same meeting: what the metrics show becomes the next change, and every change is measured again. NIST also published a dedicated Generative AI Profile (NIST AI 600-1) in July 2024 as a companion to its AI Risk Management Framework, which is useful if you need a governance structure that auditors recognise.

If you are planning AI capability across the organisation, including roles, priorities and running several agents side by side, the page on a fractional Head of AI explains how that role can be set up. Terms such as LLMOps, tracing and evals are defined in the AI glossary.

Frequently asked questions

How does AgentOps differ from MLOps and LLMOps?

MLOps covers trained models; LLMOps covers applications built around language models. AgentOps adds multi-step workflows, tool calls and write access to other systems, all of which need their own monitoring and approval rules.

How much notice do model providers give before retiring a model?

Each provider sets its own policy. Anthropic states at least 60 days' notice before retirement for publicly released models; check the terms of every provider you rely on.

Does an AI agent need a 24/7 on-call rota?

Only if the process runs around the clock and cannot wait. For most office workflows, business-hours support plus a tested fallback to manual handling is enough.

What should we measure first?

Error rate, rejection rate of agent output, run time, cost per run and run volume. These five numbers reveal most problems before users start complaining.

Sources

  1. Anthropic: Model deprecations (accessed October 2026) platform.claude.com
  2. Anthropic: Building effective agents (19 December 2024) anthropic.com
  3. Google SRE Book: Embracing Risk (error budgets) sre.google
  4. OWASP Top 10 for LLM Applications 2025 genai.owasp.org
  5. OWASP LLM06:2025 Excessive Agency genai.owasp.org
  6. OpenTelemetry: Semantic Conventions for Generative AI github.com
  7. EU AI Act, Article 26: Obligations of deployers of high-risk AI systems artificialintelligenceact.eu
  8. NIST AI Risk Management Framework nist.gov