Guardrails for AI Agents: Securing Inputs, Outputs and Actions
A chatbot only talks. An agent reads content, makes decisions and calls tools: it opens a ticket, sends a draft, updates a record. That changes what can go wrong. Guardrails are the checks and limits that stop an agent from saying or doing the wrong thing. This guide sorts them into three layers, reviews the tools available today and closes with a pre-launch review you can run against your own agent.
What are guardrails for AI agents?
Guardrails are controls placed around a language model that inspect an agent's inputs, outputs and actions, and that can block, rewrite or escalate them to a person. They exist because models are probabilistic and can be steered by any text that reaches their context window.
The key distinction is between instructions and enforcement. A line in the system prompt such as "never reveal account data" shapes behaviour, but a cleverly worded email or web page can override it. A real guardrail sits where the model cannot argue with it: in application code, in access rights, in an approval step.
The OWASP Top 10 for LLM Applications 2025 is a widely referenced risk catalogue to design against. Four entries map directly onto agents:
- LLM01:2025 Prompt Injection — input changes how the model behaves (see the prompt injection glossary entry).
- LLM05:2025 Improper Output Handling — output is passed on without validation.
- LLM06:2025 Excessive Agency — the agent can do more than its task needs.
- LLM07:2025 System Prompt Leakage — internal instructions leak to users.
Each of these risks lines up with one of the three layers described next.
How do guardrails work? The three layers
Guardrails act at three points: before the model (input), after the model (output) and before every tool call (action). Each layer catches different failures and none replaces the others. The action layer does most to cap the damage when something gets through.
Layer 1: screening inputs
Input guardrails cover everything entering the model's context — not just the user's message, but also content the agent fetches itself: emails, PDFs, web pages, search results and tool output. OWASP separates direct and indirect prompt injection: in the direct case the user's own prompt alters behaviour, in the indirect case the instructions hide in external sources such as websites or files.
Common input checks:
- Scope filters. Does the request belong to the agent's job at all? A quotation agent has no reason to write poetry or discuss politics.
- A screening model in front of the main model. Anthropic's guide to mitigating jailbreaks and prompt injection recommends a lightweight model as a pre-screen, with structured output constraining its answer to a simple classification.
- Labelling third-party content. The same guide advises placing untrusted content only in tool results, stating where it came from and JSON-encoding it so it reads unambiguously as data rather than as instructions.
- Sensitive data handling. Personal data or credentials are detected and masked before anything is sent to an external model provider.
Layer 2: validating outputs
Output guardrails treat the model's answer as untrusted input from outside. For Improper Output Handling, OWASP recommends a zero-trust stance: validate model output before it reaches backend functions, encode it for its destination, and use parameterised queries whenever it touches a database.
In practice:
- Structure checks. If the next system expects JSON with fixed fields, a schema validates every response. Anything that fails is rejected or regenerated.
- Business-rule checks. Does a draft quote contain prices that are not on the price list? Does a support answer cite a part number missing from the catalogue? Comparing output with master data often catches more real errors than any generic filter.
- Leak checks. If the answer contains fragments of the system prompt, internal URLs or secrets, it is blocked.
- Content safety. Classifiers such as Llama Guard 3 screen prompts and responses against 14 hazard categories (S1 to S14) and, according to the model card, support eight languages.
Layer 3: limiting actions
Action guardrails apply before an agent calls a tool, that is, before every act of tool use or function calling. This layer decides how much harm a mistake or a successful manipulation can cause. OWASP traces Excessive Agency to three root causes: too much functionality, too many permissions and too much autonomy over high-impact operations.
The countermeasures OWASP lists are refreshingly concrete:
- offer only the tools the agent truly needs;
- keep each tool narrow, for instance read-only, and avoid open-ended tools such as shell access or arbitrary URL fetching;
- grant minimal permissions in downstream systems and enforce them through the credentials each tool uses;
- act in the context of the requesting user rather than through an all-powerful service account;
- require human approval for high-impact actions (human-in-the-loop);
- enforce authorisation in the downstream system instead of letting the model decide whether an action is allowed.
OWASP adds that logging, monitoring and rate limiting do not prevent excessive agency but do limit the damage, so they belong to this layer as well.
Which tools exist for building guardrails?
There are three classes of tooling: guardrail features built into agent frameworks, standalone guardrail libraries, and dedicated classifier models. The right mix depends on how many agents you run and whether rules should live in one central place. Details below were checked in November 2026.
| Tool | Kind | Layers | Licence / access |
|---|---|---|---|
| OpenAI Agents SDK | Framework feature | Input, output, tool | Part of the SDK |
| NeMo Guardrails | Library | Input, retrieval, dialog, execution, output | Apache 2.0 |
| Guardrails AI | Library with validator hub | Input, output | Apache 2.0 |
| Llama Guard 3 | Classifier model | Input, output | Model card on Hugging Face |
| Your own code | Schemas, allowlists, approvals | All three | — |
Framework guardrails. The OpenAI Agents SDK provides input, output and tool guardrails. When a check sets its tripwire, the runner raises an exception and halts the run. One detail matters in production: input guardrails run in parallel with the agent by default, so the agent may already spend tokens and call tools before the tripwire stops it. Setting run_in_parallel=False makes the check blocking. The documentation also notes that tool guardrails do not cover handoffs or hosted tools.
NeMo Guardrails. NVIDIA's open-source library (documentation) distinguishes input, retrieval, dialog, execution and output rails. You configure models and rails in YAML and write flows and guardrail logic in its own language, Colang. According to its GitHub repository it is licensed under Apache 2.0, with 0.24.1 as the latest release.
Guardrails AI. This Python framework (GitHub, Apache 2.0) builds "guards" from individual validators. Validators are published on Guardrails Hub and installed one by one, covering checks such as regex matching or toxic-language detection. It can also push a model towards structured output that matches a Pydantic model.
Classifier models. Llama Guard 3 is fine-tuned from Llama 3.1 8B and returns whether content is safe and, if not, which categories it violates. Category S14 covers code-interpreter abuse in tool calls, according to the model card. Because it can run on your own infrastructure, it suits organisations that need sovereign AI setups.
Your own code. No off-the-shelf tool knows your permission model, so action checks are usually written in-house. The sketch below (Python 3.12, standard library only) shows the idea:
# Action guardrail: inspect every tool call before it runs
ALLOWED_TOOLS = {"create_quote_draft", "search_products"}
NEEDS_APPROVAL = {"send_quote"}
MAX_DISCOUNT = 0.10
def check_action(tool: str, args: dict) -> str:
if tool in NEEDS_APPROVAL:
return "approval" # hand over to a person
if tool not in ALLOWED_TOOLS:
return "blocked" # unknown tool
if args.get("discount", 0) > MAX_DISCOUNT:
return "approval" # value limit exceeded
return "ok"
The check lives in application code, not in the prompt. The model can neither see it nor talk its way past it.
Where are guardrails applied? Patterns by agent type
How strict your guardrails must be depends on what the agent is allowed to do. An agent that only reads and drafts mainly needs input and output checks. An agent that writes into ERP, CRM or email systems additionally needs hard action limits and approvals.
Knowledge agents answer questions from internal documents. The main risks are wrong or invented answers and exposing content a user should not see. Guardrails: respect document permissions at retrieval time, accept answers only with a cited source, and force an "I don't know" when no source exists. See AI agents for knowledge management.
Drafting agents prepare quotes, replies or reports that a person sends. The output layer is where the value lies: check prices, part numbers and lead times against master data. AI-assisted quotation is a typical case, where every draft is validated before it goes out.
Acting agents open tickets, book appointments, change records or send messages externally. Here the action layer is decisive: narrow tools, the requesting user's permissions, value limits and approvals. In technical service, that might mean the agent proposes a field visit but cannot dispatch an engineer by itself.
Agents with web or inbox access are the most exposed to indirect prompt injection because they read untrusted text all day. Label external content as data, screen tool results before acting on them, and avoid giving them any tool that could quietly send data outside the organisation.
What do you need in place first?
Guardrails assume you know what the agent should do, what data it may see and who owns each decision. Without that clarity every check stays vague. Technically, you need control over tool calls, a permission model in downstream systems, and logging.
Organisation:
- a written description of the agent's job, including explicit exclusions;
- a list of consequential actions, each with a named person who approves them;
- a rule for who is notified when requests are blocked, and how quickly.
Data and systems:
- master data to validate output against, such as price lists, product catalogues and account status;
- roles and permissions in ERP, CRM or document stores that the agent can inherit, instead of one master key;
- a log of inputs, outputs, tool calls and check results, which can be operated in line with the GDPR when retention periods and access are defined.
Technology:
- an architecture in which every tool call passes through your own code before it executes;
- an automated suite of attack cases and edge cases that runs against every new version.
How do you roll guardrails out?
Work backwards from actions to inputs. First decide the worst the agent could do, then which outputs would be harmful, and only then how to filter inputs. That way you invest first where damage occurs, rather than in filters that can be bypassed.
- Inventory the actions. List every tool with its read or write access, target system and potential harm. Remove anything without a clear need.
- Trim permissions. Give each tool the narrowest rights in the target system, ideally on behalf of the requesting user.
- Define approvals. Anything that leaves the organisation, moves money or deletes data gets an approval step with value thresholds.
- Validate outputs. Define schemas, add master-data checks, and scan for leaked prompts and secrets.
- Screen inputs. Add scope filters, label external content and, where needed, put a classifier in front of the main model.
- Attack your own agent. Build a collection of manipulated emails, documents and requests and run it before each release. Anthropic's guide explicitly recommends red-teaming agents with injected documents, emails and tool output before deployment.
- Watch production. Review blocked requests, approval rates and anomalies regularly, and tighten the rules. The guide to AgentOps covers how this fits into day-to-day operations, and evals make the tests repeatable.
Pre-launch review
Before an agent goes live, you should be able to answer yes to each of these:
- Is every tool justified, and are open-ended tools (shell, arbitrary URLs, free-form SQL) excluded?
- Does every write call run with minimal rights, enforced by the target system itself?
- Does every consequential action have an approval step and a named owner?
- Is output checked against schema and master data before any system accepts it?
- Is external content clearly marked as data inside the context?
- Are input checks blocking wherever an early start could already cause harm?
- Is there an attack test suite that passed and reruns on every change?
- Is every decision logged, and does someone actually read the log?
- Is there a kill switch that stops the agent immediately without halting other systems?
What are the limits of guardrails?
Guardrails reduce risk; they do not remove it. On prompt injection, OWASP states it is unclear whether fool-proof prevention exists. Any model-based filter can be wrong, and rules that are too strict make the agent useless. The work is in balancing protection against usefulness.
False positives. An oversensitive filter blocks legitimate work. Staff then route around the agent or stop trusting it. Track how often a block was unjustified, not just how often blocks happen.
False negatives. Classifiers miss novel attack patterns. If input filters are your only defence, you have nothing left once one slips through. That is why the action layer is the foundation.
Latency and cost. Every check that calls a second model costs time and tokens. Running checks in parallel saves time but means the agent may already be working when the check fires, a trade-off the OpenAI Agents SDK documents directly.
False comfort. A dashboard full of green checks feels reassuring but says nothing about untested cases. Guardrails without regular attack testing go stale.
Approval fatigue. People who click through hundreds of approvals a day stop reviewing them. Keep approvals for the few genuinely consequential actions and show the information needed to decide.
Legal context. Guardrails help with privacy and traceability requirements but do not replace legal review. This guide is orientation, not legal advice.
How should you get started?
Start with one agent and its tool list. Remove every tool it does not need, put an approval step on write actions, and validate outputs against a schema. These three steps take little effort and target the places where mistakes have consequences, well before any framework is necessary.
In our own operations, the approach that has held up is to design guardrails into the architecture from day one rather than bolting them on later: every tool call passes through one central check function, every check writes to the log, and new tools receive permissions only once their potential for harm has been written down.
From there, extend step by step: add an input classifier once the agent reads open text from outside, adopt a library such as NeMo Guardrails or Guardrails AI when several agents need to share the same rules, and automate attack testing as soon as the agent is updated regularly.
For an overview of which agents fit which business processes and how they connect to existing systems, see AI agents for business. More articles on architecture and operations are collected in the guides section.
Frequently asked questions
Is a well-written system prompt enough of a guardrail?
No. A system prompt nudges the model but enforces nothing. Controls that must hold belong in code, permissions and approval steps outside the model.
Do I need a dedicated guardrail framework?
Not necessarily. Schema validation, a tool allowlist and an approval step cover many agents. Libraries such as NeMo Guardrails or Guardrails AI pay off when several agents share a central set of checks.
Will guardrails slow my agent down?
Checks that call a second model add latency and token cost. Deterministic checks like schemas and rules add very little, so keep the expensive checks for the places where the risk actually sits.
Can guardrails stop prompt injection completely?
No. OWASP itself states it is unclear whether fool-proof prevention methods exist. The realistic goal is to keep the damage of a successful injection small.
Sources
- OWASP Top 10 for LLM Applications 2025 genai.owasp.org
- OWASP LLM01:2025 Prompt Injection genai.owasp.org
- OWASP LLM05:2025 Improper Output Handling genai.owasp.org
- OWASP LLM06:2025 Excessive Agency genai.owasp.org
- Anthropic: Mitigate jailbreaks and prompt injections platform.claude.com
- OpenAI Agents SDK: Guardrails openai.github.io
- NVIDIA NeMo Guardrails documentation docs.nvidia.com
- NeMo Guardrails on GitHub github.com
- Guardrails AI on GitHub github.com
- Llama Guard 3 model card huggingface.co