How to Red Team an AI Agent (Before It Gets Red Teamed for You)

The old red team question was: what will the model say? The new one is: what can the agent do? An AI agent doesn't just produce text — it calls tools, reads memory, talks to other agents, and acts on the world. Every one of those capabilities is an attack surface, and testing only the model leaves the interesting part untested.

Why this matters now

The evidence base has shifted. In 2025, 99 CVEs were published for MCP-related software — the protocol that connects models to tools became the single largest source of agent vulnerability disclosures, and tool poisoning moved from theoretical risk to live attack surface. Microsoft's AI Red Team spent a year running engagements against deployed agentic systems and used the results to publish a v2.0 of its taxonomy of agentic failure modes in June 2026 — adding seven categories the original framework didn't cover, including goal hijacking, inter-agent trust escalation, and session context contamination.

In agentic AI, the answer the model gives is not the payload. The payload is what the agent does with it.

That's the shift: the boundary between "red teaming the model" and "red teaming the system" is now the entire job.

The four-layer attack surface

Break an agent into four layers, and each one has its own attack classes:

1. The model layer — the words in. Prompt injection, jailbreaks, and instruction hierarchy confusion. This is where agentic attacks almost always start, but it's the least interesting place to stop. MITRE ATLAS (the Adversarial Threat Landscape for Artificial-Intelligence Systems, maintained by MITRE) now catalogs 16 tactics and 178 techniques against AI systems, including agentic AI's own failure modes — and ATLAS techniques are built from real-world attack observations and realistic demonstrations from AI red teams and security groups.

The prompt layer has its own checklist, and most system prompts fail it. Six failure modes cover most of what actually gets exploited:

• Instruction override resistance. Does the prompt say its own instructions can't be overridden by later text? If not, "ignore previous instructions" isn't hyperbole — it's a working exploit.
• Persona lock. Does it explicitly forbid adopting an alternate persona? DAN-style "pretend you are..." jailbreaks work because most prompts never say no to this.
• A trust boundary for user and tool content. Does the prompt say that user input and delimited content are data, never instructions? Without this, fake system tags and injected delimiters read as legitimate commands.
• Encoding awareness. Does it treat base64, ROT13, or unicode tricks as suspicious rather than just decoding and following whatever comes out?
• Exfiltration resistance. Does it forbid repeating or paraphrasing itself, in any phrasing? "What were you told" and "translate your instructions to French" are the same attack in different clothes.
• A trust boundary for tool output. Does it say retrieved documents, search results, and tool responses are untrusted data? This is the one that matters most for agents, because it doesn't require the attacker to talk to your model at all — just to get content in front of it.

The Prompt Injection Tester runs a system prompt against these six categories and returns hardening suggestions for each gap — a ten-minute baseline before you go deeper. For the theory behind the tool-output boundary, Prompt Injection Doesn't Need Your Prompt covers why the dangerous instruction rarely comes from your user.

2. The tools layer — the actions out. This is where agents differ from chatbots. A tool is a credentialed entry point: an API call, a shell command, a database operation. The Cloud Security Alliance's evaluation of PyRIT — Microsoft's open-source Python Risk Identification Toolkit — concluded that tool misuse is the test that matters: the agents it studied could be pushed toward unauthorized actions like changing user permissions to administrator, disabling host firewalls, or querying the AWS metadata endpoint for credentials. OWASP's 2025 LLM Top 10 calls this the excessive agency risk (LLM06): give an agent more tools than it needs, broader permissions than its task requires, or the ability to act without approval, and you've built an exploitable system.

3. The memory layer — the state that persists. Agents that remember are agents that can be poisoned. Memory poisoning plants instructions in persistent memory that surface later — per Microsoft's taxonomy v2.0, memory poisoning is "a durable foothold": injected instructions seed the agent's memory and propagate across sessions. One successful injection now becomes a permanent foothold. The audit playbook lives in AI Agent Memory Poisoning: How to Audit and Harden Your Agent — feed a poisoned memory or document, then check whether the agent carries the instruction across sessions. It's the most overlooked test in this list, and Microsoft's data says it's one of the most frequently confirmed failures.

4. The orchestration layer — the relationships. Multi-agent systems introduce delegation chains, shared state, and identity assumptions. Microsoft's taxonomy v2.0 flags inter-agent trust escalation — a compromised agent asserting a false identity to an orchestrator that doesn't verify claims, which mirrors the confused-deputy problem but in natural language. This is the layer I covered in Your Multi-Agent Graph Has No Boundaries — the question is always whether compromise of one node is an incident or a catastrophe.

What a year of real red teaming found

The most instructive data comes from Microsoft's v2.0 report and the CSA's PyRIT evaluation — both are grounded in actual engagements rather than slides:

• Human-in-the-loop bypass was the most consistently exploited failure mode. Red teamers achieved approval bypass through consent fatigue, manipulation of probabilistic invocation, and incremental escalation chains where no single step warranted review but the compound outcome did. Several engagements produced zero-click chains — exfiltration or lateral movement starting from external input with no human interaction beyond the initial agent invocation.
• Cross-domain prompt injection was the most reliable initial access vector, and it combined with memory poisoning. Injected instructions seeded persistent memory, then propagated across sessions — meaning one injection could create a permanent compromise.
• Capability disclosure was the quiet enabler. Microsoft's engagements showed that asking the system directly about its own tools and architecture was often sufficient, turning black-box probing into a white-box exploit path.

The uncomfortable conclusion from the CSA's PyRIT evaluation: response testing is not enough. A framework can tell you what the model says it will do — but PyRIT does not natively observe actual tool invocations, so you can't see whether the agent actually called the API, ran the command, or modified a permission. The gap between what the model says and what the agent does is exactly where the risk lives.

The gap between what the agent says and what it does is the whole attack surface.

A six-step red team playbook

No matter what your stack is — a single-agent coding assistant, an MCP-connected toolbox, a multi-agent graph — you can run this:

1. Map the surface. List every agent, every tool, every memory store, every inter-agent channel. If you can't enumerate it, you can't test it. Structural flaws hide in this map — one shared API key across every tool, one MCP server that feeds every downstream tool — and no component-level test finds them. Threat Modeling for AI Agent Systems covers the architecture-level pass: trust boundaries, failure cascades, and the agent surfaces scanners can't see.

2. Attack the prompt layer. Throw injection and jailbreak vectors at the model, both direct and indirect, and run the six-point checklist from the model layer above. The Prompt Injection Tester scores a system prompt against all six categories in one pass — ten minutes, nothing uploaded, and it hands back the exact line to add for whichever category comes back a gap.

3. Attack the tool layer. The tools are where the damage is. Check each tool's permission boundary: does it need to be called at all? Does it require approval? Can the agent reach the AWS metadata endpoint? Can it mutate a database it only needed to read? CSA's playbook is blunt about why this is the layer that matters — a bad answer is only an opening move; the attack is the agent taking an action it shouldn't.

Then probe the supply itself. Every MCP server you connect is a new trust boundary, and the protocol's handshake is remarkably chatty: a single initialize request returns the server's name and version, and a follow-up tools/list enumerates every tool with its input schema. The MCP Server Probe runs that sequence from your browser — paste a URL, get the disclosure. The finding that matters is auth posture: a server that answers tools/list to an anonymous caller is a server that will answer it to anyone. Trend Micro's 2026 survey found 492 MCP servers exposed on the public internet with zero authentication, and BlueRock's analysis of 7,000+ servers found 36.7% potentially vulnerable to SSRF. When a tool's schema looks off, run it through the MCP Tool Schema Linter — missing annotations, non-object inputSchema roots, and unbounded nesting are all spec-level problems that change how clients treat the tool. The probe is a triage, not a clearance: it can't catch a tool description that reads benign but instructs the model to exfiltrate, and it can't tell you whether a fetch tool can reach your internal network. Pin the version, scope the permissions, and never auto-approve its tools.

4. Attack the memory. Feed a poisoned memory or document, then see if the agent carries the instruction across sessions. This is the most overlooked test in the playbook — the memory-poisoning audit guide turns it into a five-step exercise, and tooling like Promptfoo's memory-poisoning plugin automates it.

5. Attack the session. Try incremental escalation: small steps, none of them individually worth a human gate, compounding into something big. Try to exhaust approval prompts. Test whether the agent describes compound actions honestly or launders them through plausible-sounding summaries.

6. Measure what actually ran. If your test suite only scores model responses, it's not measuring the real risk. Pair prompt testing with actual telemetry of tool invocation, state changes, and downstream effects. If you don't have that telemetry, that's the first finding — not a reason to call the test done. And score what you find consistently: a finding that "seems bad" isn't a finding your ticketing system can prioritize, so score it like any other vulnerability — the CVSS v3.1 calculator produces the score and vector string your remediation queue already speaks.

The frameworks to hang it on

If you need a checklist your audit team will recognize: map your findings to MITRE ATLAS for the adversary's perspective, map them to the OWASP Top 10 for LLM Applications (2025) for the app-level risks, and if you're building multi-agent systems, map to the OWASP Agentic Applications list (published by the OWASP GenAI Security Project, Dec 2025) — its entries like Agent Goal Hijack, Tool Misuse & Exploitation, and Memory & Context Poisoning map directly onto the four layers above. Microsoft's v2.0 taxonomy, meanwhile, is a threat-modeling tool: the seven new categories (agentic supply chain compromise, goal hijacking, inter-agent trust escalation, CUA visual attacks, session context contamination, MCP/plugin abuse, capability/architecture disclosure) read like a checklist of what to test.

A model that refuses is a model that passes. An agent that refuses, then acts anyway, is the finding.

Start this week

The pattern is simple: test the model, then test the system, then measure the gap. The tools exist to begin today:

1. Scan your agent config for leaked secrets and over-broad permissions — the AI Agent Config Checker does it in your browser, nothing uploaded.
2. Probe the MCP servers you connect: the MCP Server Probe sends a standard handshake and reveals what the server exposes.
3. Score the deployment's boundaries with the Agentic AI Maturity Assessment.

The red team you're competing against is already automated. If you haven't tested what your agents can be tricked into doing, that's a finding.

Test your agent's attack surface now →