Skip to content

AI Agent Security: Threats and Countermeasures

A chatbot can write a wrong sentence. An AI agent that reads inboxes, queries the ERP and opens tickets can do a wrong thing. That shifts the security question away from what the model says and towards permissions, data flows and actions. This guide is written for IT leads who have to introduce or assess agents without an AI security specialist on the team.

What does security mean for AI agents?

Agent security means that even if the language model is manipulated, the agent cannot leak data, borrow privileges it should not have, or carry out harmful actions. You treat the model itself as an untrusted component and enforce safety in its surroundings: permissions, network, approvals and logs.

What separates this from ordinary application security is a property of language models that no setting can switch off. Germany's Federal Office for Information Security (BSI) puts it plainly in its baseline guidance on indirect prompt injection (version 1.0, September 2026): an LLM cannot, on its own, tell user instructions, system instructions, external instructions and data apart; it sees one stream of tokens. To the model, an invoice attached as a PDF is essentially the same kind of text as the request "summarise this invoice".

Three principles follow, and they run through everything below:

  • The model never decides on authorisation. Whether an action is allowed is checked by the target system, not by the prompt.
  • Every capability is attack surface. Each tool, write permission and network route widens what an attacker can reach through the model.
  • Containment beats detection. Filters catch many attacks, not all. Build so that an attack that gets through can do little harm.

The main industry catalogues reflect this. The OWASP Top 10 for LLM Applications 2025 lists prompt injection first (LLM01) and "Excessive Agency" – too much functionality, permission or autonomy – as LLM06. In December 2025 OWASP added a list just for agents, the OWASP Top 10 for Agentic Applications for 2026.

What does an agent threat model look like?

A workable threat model for agents has four classes. Injection: outside instructions reach the model. Exfiltration: data leaves the organisation. Identity: the agent acts with too much or someone else's authority. Execution: the agent runs code or actions nobody intended. Most real-world attacks chain at least two of these.

Each class maps onto one part of an agent – inputs, model, tools, outputs:

Threat classWhere it entersTypical exampleOWASP reference
InjectionInputs: email, documents, web pages, tool resultsHidden white-on-white instruction inside a PDFLLM01, ASI01 Agent Goal Hijack
ExfiltrationOutputs: replies, links, emails, API callsA markdown image whose URL carries confidential dataLLM02 Sensitive Information Disclosure
IdentityCredentials, tokens, service accountsAgent uses a shared admin account instead of the user's rightsLLM06, ASI03 Identity & Privilege Abuse
ExecutionTools, shell, scripts, workflowsAgent runs an injected command in a terminalASI02 Tool Misuse & Exploitation, ASI05 Unexpected Code Execution

The agentic OWASP list also names risks that cut across these classes: ASI04 Agentic Supply Chain Vulnerabilities, ASI06 Memory & Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents. For planning purposes, read them as amplifiers. Poisoned memory is a delayed injection; a tampered plugin is execution with someone else's code.

Injection: outside instructions in the context

In direct prompt injection the attacker types the instruction themselves. The more serious case for organisations is indirect injection, where the instruction sits inside content the agent processes on behalf of a legitimate user. The BSI points out that carriers include not only visible text but also metadata, annotations and comments, using tricks such as invisible Unicode characters or white text on a white background. For an agent that triages supplier emails, every incoming message is a possible attack channel.

Exfiltration: data leaving the building

Most successful attacks are after data, and the route out is often inconspicuous. The BSI describes exfiltration links: an embedded image is loaded from an external server, and the data travels in the image URL – no click required. Other channels include outgoing email, webhooks, search queries and writes to publicly reachable storage.

Identity: whose permissions is the agent using?

Many agents run under a single technical account shared by every user, often with generous rights for convenience. One injection is then enough to read data the requesting user could never see. Under LLM06 Excessive Agency, OWASP names three root causes: unnecessary functionality, excessive permissions and too much autonomy over high-impact actions.

Execution: actions nobody asked for

Agents with shell access, code interpreters or generic HTTP tools can actually run injected commands. This is not limited to coding assistants. A workflow agent that "only" opens tickets and places purchase orders still takes actions with real consequences.

When does an agent become dangerous?

An agent becomes genuinely dangerous when it has three properties at once: it processes untrusted content, it can reach sensitive data or systems, and it can communicate externally or change state. Remove any one of the three and the damage a successful attack can cause shrinks considerably.

This idea is now widely shared. Simon Willison described it in June 2025 as the "lethal trifecta": access to private data, exposure to untrusted content, and the ability to communicate externally. In October 2025 Meta turned it into the "Agents Rule of Two": until prompt injection can be reliably detected and refused, an agent should satisfy no more than two of the three properties within a single session – processing untrustworthy inputs, accessing sensitive systems or private data, and changing state or communicating externally. The BSI's guidance on evasion attacks lists the same factors as drivers of indirect attacks.

For an IT lead, this is the most useful test in the whole field, because it needs no AI expertise. Three examples:

  • An agent summarises inbound customer emails and saves a reply draft to the drafts folder. Untrusted content: yes. Sensitive data: limited. External communication: no, a person sends the reply. Two out of three – acceptable as long as drafts are never sent automatically.
  • An agent researches the web and writes findings to the internal wiki. Untrusted content: yes. State change: yes. Sensitive data: should be no. If the wiki account can also read HR pages, the third property is quietly met.
  • An agent reads supplier emails, checks prices in the ERP and sends confirmations on its own. All three. This design needs either a human before sending or a split into two agents with separate permissions.

Which countermeasures actually work?

Effective protection is layered and addresses all four threat classes: clean and label inputs, minimise permissions, filter outputs, require approval for high-impact actions, and log everything. Most of these controls are standard IT security, applied to a new kind of component.

The list below groups controls from the BSI publications, the OWASP lists and the Model Context Protocol (MCP) security documentation by threat class.

Against injection

  • Sanitise inputs: strip or flag invisible characters, hidden text layers, comments and metadata before content reaches the model. Input sanitisation is the first of the BSI's five baseline measures.
  • Separate trust zones: mark outside content in the prompt clearly as data, distinct from the user's request.
  • Harden the system prompt: explicit priorities, a defined role, and a reminder of the core instruction after the untrusted content. Useful, but never the only layer.

Against exfiltration

  • Filter outputs: the BSI's egress filter inspects the finished answer before it reaches the user, including embedded images and links to non-approved addresses. It recommends an allowlist of permitted destinations and blocking when in doubt.
  • Restrict the network: the agent should reach only the endpoints it needs. An egress proxy with an allowlist does more than any rule in the prompt.
  • Canary tokens: a secret marker in the system prompt that must never appear in a response signals a leak when it does.

Against identity abuse

  • Act as the user: actions run with the permissions of the person who triggered them, not a shared account. OWASP recommends exactly this under LLM06, and the BSI calls it model action privilege minimisation.
  • Never pass tokens through: the MCP security best practices explicitly forbid token passthrough – an MCP server must not accept tokens that were not issued for it.
  • Grant scopes progressively: start with a minimal set and request more only when an action needs it. Broad scopes such as files:* or admin:* turn a stolen token into a much bigger incident.

Against unintended execution

  • Narrow the tools: a "write file to folder X" tool instead of a shell; read-only database access instead of full access. OWASP calls this "least agency".
  • Sandbox: process code, documents and web content in isolated environments so a successful attack stays contained.
  • Human approval: a person confirms high-impact actions such as sending, paying, deleting or changing permissions. The BSI stresses that approvers need training to spot manipulated proposals.
  • Vet local servers: before a local MCP server is connected, the client should show the exact launch command and ask for explicit consent; servers should run sandboxed with minimal privileges.

Across all classes: complete logging of prompts, tool calls and results so incidents can be reconstructed, plus monitoring for unusual patterns.

How guardrails are actually built – input, output and action checks in detail – is covered in Guardrails for AI Agents. The difference: guardrails are the toolkit; the threat model on this page tells you which tool belongs where.

What do you need in place before deploying agents securely?

Secure agent operations rest on foundations most organisations already have, or should: a clean permission model with personal accounts, network segmentation with controlled egress, central logging, and an approval process for new software. Without that base, even a well-designed agent cannot be secured.

Data: before an agent touches a repository, be clear about who can see what there today. Overly broad sharing on SharePoint or file servers is a common finding; the agent inherits it and makes it searchable in plain language.

Systems: target systems must enforce permissions themselves. An ERP that answers every API call with full rights cannot be restricted by an instruction in a prompt. Systems with per-task service accounts, OAuth scopes and audit logs are a good fit.

Organisation: agents need the same ownership as any other application: an inventory, a named owner, a permissions overview and a way to switch them off. Approvers need brief training in recognising manipulated proposals.

Hosting: where the model runs is a separate decision. Self-hosted or sovereign operation settles which data leaves the organisation and who can access prompts and logs; Local AI for Business covers what that involves. It does not protect against injection, because a local model reads the poisoned email too.

Regulation: for high-risk AI systems, Article 15 of the EU AI Act requires resilience against attempts by unauthorised third parties to alter use, outputs or performance by exploiting vulnerabilities, naming data poisoning, model poisoning and adversarial inputs explicitly. Whether a given agent is high-risk depends on its purpose. Treat this as orientation, not legal advice; outside the EU, check your own sector rules.

How do you secure agents step by step?

Securing agents follows a repeatable sequence: inventory them, check each against the three dangerous properties, cut back permissions and tools, add approvals and filters, log, and test with real attacks. It works the same for one pilot agent as for twenty in production.

  1. Build an inventory. Which agents, assistants and LLM-based automations already run? Include copilot features in office software and tools individual teams set up themselves.
  2. Sketch each agent's data flow. Inputs (email, upload, web, tool results), data, tools, outputs. One page per agent is enough.
  3. Check the dangerous combination. Does the agent have all three properties? If so, remove one, split the agent, or put a human approval in front of the external effect.
  4. Decide the identity. User's permissions or a service account? If a service account, cut it to what is needed and review it regularly. Never forward tokens between services.
  5. Trim the tools. Replace generic tools (shell, arbitrary HTTP, full database access) with narrow, purpose-built ones. Remove tools nobody uses.
  6. Lock down outputs. Allowlist network destinations, filter embedded images and unknown links, never render external content automatically.
  7. Define approvals. Rule of thumb: anything that leaves the organisation, moves money, deletes data or changes permissions.
  8. Log. Store prompts, tool calls, parameters and results centrally, with retention rules and access control.
  9. Attack it. Before go-live and after major changes, test with crafted documents, emails carrying hidden instructions and attempts to extract the system prompt. The BSI is clear that its baseline measures do not replace application-specific red teaming.

Step 3 has the biggest leverage: an agent that never reaches the dangerous combination needs far less protection in every step after it.

What are the limits of these countermeasures?

None of these controls stops prompt injection completely. The BSI notes that even purpose-built defences can be bypassed by adaptive attacks. Security therefore means limiting the possible damage and detecting incidents, not ruling attacks out.

That leads to a few honest limitations worth knowing:

  • Detection filters are probabilistic. Classifiers and keyword lists catch known patterns. Rephrased, encoded or image-embedded instructions get through.
  • Human approval wears thin. Someone confirming a hundred agent proposals a day eventually stops reading them. OWASP lists the exploitation of that trust as its own risk (ASI09). Reserve approvals for actions with real consequences.
  • Chained agents multiply errors. When one agent hands its output to the next, an injection travels along. Every agent in a chain must treat upstream output as untrusted.
  • Long-term memory preserves attacks. Once something lands in an agent's memory, it affects later sessions. Memory needs provenance and expiry.
  • The supply chain is hard to see. MCP servers, plugins and agent skills from public sources often run with the user's privileges. They need the same vetting as any other software, which takes real effort.
  • Security costs convenience. Fewer permissions and more approvals make agents slower and less autonomous. Make that trade-off deliberately.

When is an agent the wrong choice despite all of this? When a task needs all three dangerous properties, cannot tolerate a human approval step, and a mistake would be severe. A conventional, rule-based workflow is often the safer answer there.

How should you get started?

Start with an inventory and a single, low-risk agent: one that reads outside content but only produces drafts. Use it to try out your permission model, logging and approval process before adding agents that act on the outside world.

A pragmatic start in three stages:

  1. This week: list every agent and AI feature in use and answer three questions for each. Does it read outside content? Can it see sensitive data? Can it act externally?
  2. This month: for every agent with three yeses, decide – split it, remove permissions or add an approval. Turn on logging for all agents.
  3. This quarter: run a structured test with crafted documents and emails, using the BSI baseline measures as a checklist.

If you want agents designed from the outset so that data, models and logs stay under your own control, the building blocks are described under sovereign AI. For typical places where agents are deployed – and good material to practise this threat model on – see AI agents.

Sources checked: November 2026.

Frequently asked questions

Can a well-written system prompt stop prompt injection?

No. Germany's Federal Office for Information Security (BSI) notes that no reliable countermeasure fully eliminates the problem. A hardened system prompt is one layer; limiting the damage has to happen in the architecture.

Does running the model on our own servers make agents secure?

It settles where data is processed and who can see prompts and logs. It does not stop injected instructions, because a self-hosted model reads the poisoned email just like a cloud model would.

Which agents should we review first?

Any agent that processes outside content, can reach internal data, and can send, write or call external services. That combination is behind most data-leak scenarios.

Do we need a dedicated AI security team?

Not to get started. The most effective controls are standard IT security: permissions, network egress rules, approval steps and logging. For high-stakes deployments, add application-specific red teaming.

Sources

  1. OWASP Top 10 for LLM Applications 2025 genai.owasp.org
  2. OWASP LLM06:2025 Excessive Agency genai.owasp.org
  3. OWASP Top 10 for Agentic Applications for 2026 genai.owasp.org
  4. Simon Willison: The lethal trifecta for AI agents simonwillison.net
  5. Meta AI: Agents Rule of Two ai.meta.com
  6. Model Context Protocol: Security Best Practices modelcontextprotocol.io
  7. BSI: Baseline protection against indirect prompt injection in document-based LLM chats (German) bsi.bund.de
  8. BSI: Evasion attacks on LLMs – countermeasures in practice (German edition) bsi.bund.de
  9. EU AI Act, Article 15: Accuracy, robustness and cybersecurity artificialintelligenceact.eu