// Field note
How to Red Team an AI Agent (Before It Gets Red Teamed for You)
The old red team question was: what will the model say? The new one is: what can the agent do? An AI agent doesn't just produce text — it calls tools, reads memory, talks to other agents, and acts on the world. Every one of those capabilities is an attack surface, and testing only the model leaves the interesting part untested.
Why this matters now
The evidence base has shifted. In 2025, 99 CVEs were published for MCP-related software — the protocol that connects models to tools became the single largest source of agent vulnerability disclosures, and tool poisoning moved from theoretical risk to live attack surface. Microsoft's AI Red Team spent a year running engagements against deployed agentic systems and used the results to publish a v2.0 of its taxonomy of agentic failure modes in June 2026 — adding seven categories the original framework didn't cover, including goal hijacking, inter-agent trust escalation, and session context contamination.
In agentic AI, the answer the model gives is not the payload. The payload is what the agent does with it.
That's the shift: the boundary between "red teaming the model" and "red teaming the system" is now the entire job.
The four-layer attack surface
Break an agent into four layers, and each one has its own attack classes:
1. The model layer — the words in. Prompt injection, jailbreaks, and instruction hierarchy confusion. This is where agentic attacks almost always start, but it's the least interesting place to stop. MITRE ATLAS (the Adversarial Threat Landscape for Artificial-Intelligence Systems, maintained by MITRE) now catalogs 16 tactics and 178 techniques against AI systems, including agentic AI's own failure modes — and ATLAS techniques are built from real-world attack observations and realistic demonstrations from AI red teams and security groups.
2. The tools layer — the actions out. This is where agents differ from chatbots. A tool is a credentialed entry point: an API call, a shell command, a database operation. The Cloud Security Alliance's evaluation of PyRIT — Microsoft's open-source Python Risk Identification Toolkit — concluded that tool misuse is the test that matters: the agents it studied could be pushed toward unauthorized actions like changing user permissions to administrator, disabling host firewalls, or querying the AWS metadata endpoint for credentials. OWASP's 2025 LLM Top 10 calls this the excessive agency risk (LLM06): give an agent more tools than it needs, broader permissions than its task requires, or the ability to act without approval, and you've built an exploitable system.
3. The memory layer — the state that persists. Agents that remember are agents that can be poisoned. Memory poisoning plants instructions in persistent memory that surface later — per Microsoft's taxonomy v2.0, memory poisoning is "a durable foothold": injected instructions seed the agent's memory and propagate across sessions. One successful injection now becomes a permanent foothold.
4. The orchestration layer — the relationships. Multi-agent systems introduce delegation chains, shared state, and identity assumptions. Microsoft's taxonomy v2.0 flags inter-agent trust escalation — a compromised agent asserting a false identity to an orchestrator that doesn't verify claims, which mirrors the confused-deputy problem but in natural language. This is the layer I covered in Your Multi-Agent Graph Has No Boundaries — the question is always whether compromise of one node is an incident or a catastrophe.
What a year of real red teaming found
The most instructive data comes from Microsoft's v2.0 report and the CSA's PyRIT evaluation — both are grounded in actual engagements rather than slides:
- Human-in-the-loop bypass was the most consistently exploited failure mode. Red teamers achieved approval bypass through consent fatigue, manipulation of probabilistic invocation, and incremental escalation chains where no single step warranted review but the compound outcome did. Several engagements produced zero-click chains — exfiltration or lateral movement starting from external input with no human interaction beyond the initial agent invocation.
- Cross-domain prompt injection was the most reliable initial access vector, and it combined with memory poisoning. Injected instructions seeded persistent memory, then propagated across sessions — meaning one injection could create a permanent compromise.
- Capability disclosure was the quiet enabler. Microsoft's engagements showed that asking the system directly about its own tools and architecture was often sufficient, turning black-box probing into a white-box exploit path.
The uncomfortable conclusion from the CSA's PyRIT evaluation: response testing is not enough. A framework can tell you what the model says it will do — but PyRIT does not natively observe actual tool invocations, so you can't see whether the agent actually called the API, ran the command, or modified a permission. The gap between what the model says and what the agent does is exactly where the risk lives.
The gap between what the agent says and what it does is the whole attack surface.
A six-step red team playbook
No matter what your stack is — a single-agent coding assistant, an MCP-connected toolbox, a multi-agent graph — you can run this:
1. Map the surface. List every agent, every tool, every memory store, every inter-agent channel. If you can't enumerate it, you can't test it.
2. Attack the prompt layer. Throw injection and jailbreak vectors at the model, both direct and indirect. The Prompt Injection Tester runs a system prompt against six known technique categories and returns hardening suggestions for each gap — a ten-minute baseline that surfaces the obvious issues before you go deeper.
3. Attack the tool layer. The tools are where the damage is. Check each tool's permission boundary: does it need to be called at all? Does it require approval? Can the agent reach the AWS metadata endpoint? Can it mutate a database it only needed to read? CSA's playbook is blunt about why this is the layer that matters — a bad answer is only an opening move; the attack is the agent taking an action it shouldn't.
4. Attack the memory. Feed a poisoned memory or document, then see if the agent carries the instruction across sessions. This is the most overlooked test, and Microsoft's data says it's one of the most frequently confirmed failures.
5. Attack the session. Try incremental escalation: small steps, none of them individually worth a human gate, compounding into something big. Try to exhaust approval prompts. Test whether the agent describes compound actions honestly or launders them through plausible-sounding summaries.
6. Measure what actually ran. If your test suite only scores model responses, it's not measuring the real risk. Pair prompt testing with actual telemetry of tool invocation, state changes, and downstream effects. If you don't have that telemetry, that's the first finding — not a reason to call the test done.
The frameworks to hang it on
If you need a checklist your audit team will recognize: map your findings to MITRE ATLAS for the adversary's perspective, map them to the OWASP Top 10 for LLM Applications (2025) for the app-level risks, and if you're building multi-agent systems, map to the OWASP Agentic Applications list (published by the OWASP GenAI Security Project, Dec 2025) — its entries like Agent Goal Hijack, Tool Misuse & Exploitation, and Memory & Context Poisoning map directly onto the four layers above. Microsoft's v2.0 taxonomy, meanwhile, is a threat-modeling tool: the seven new categories (agentic supply chain compromise, goal hijacking, inter-agent trust escalation, CUA visual attacks, session context contamination, MCP/plugin abuse, capability/architecture disclosure) read like a checklist of what to test.
A model that refuses is a model that passes. An agent that refuses, then acts anyway, is the finding.
Start this week
The pattern is simple: test the model, then test the system, then measure the gap. The tools exist to begin today:
- Scan your agent config for leaked secrets and over-broad permissions — the AI Agent Config Checker does it in your browser, nothing uploaded.
- Probe the MCP servers you connect: the MCP Server Probe sends a standard handshake and reveals what the server exposes.
- Score the deployment's boundaries with the Agentic AI Maturity Assessment.
The red team you're competing against is already automated. If you haven't tested what your agents can be tricked into doing, that's a finding.