What is AI red teaming and how is it different from traditional penetration testing?

Traditional penetration testing attacks code and infrastructure. AI red teaming attacks behaviour. There is often no memory corruption and no malformed packet. The attack is a sentence in a document that the agent reads and treats as an instruction. OWASP's GenAI Red Teaming Guide splits the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. You still need the traditional testing. You now need this on top of it.

How do you red team an LLM application or AI agent?

Start from the OWASP Top 10 for Agentic Applications, published in December 2025. Work through goal hijack, tool misuse, identity and privilege abuse, memory and context poisoning, insecure inter agent communication and rogue agents. For each one, try the attack rather than reasoning about it. Plant an instruction in content the agent reads. Try to make it call a tool it should not. Try to read another tenant's data through the vector store. Then repeat after every prompt change, tool addition or model upgrade.

What is indirect prompt injection and how do you defend against it?

Indirect prompt injection is when an attacker hides an instruction inside content the agent reads later, such as a web page, a support ticket or a PDF. Unit 42 documented 22 distinct payload engineering techniques in the wild in March 2026, with 37.8 percent using plain visible text and 85.2 percent relying on social engineering framing. Defence is layered: treat all retrieved content as untrusted input, scope tool permissions tightly, require confirmation for irreversible actions, log every tool call, and test continuously.

Does the EU AI Act require red teaming or adversarial testing?

For general purpose AI models with systemic risk, yes. Article 55(1)(a) requires providers to perform model evaluation using standardised protocols, including conducting and documenting adversarial testing to identify and mitigate systemic risks. Those obligations applied from 2 August 2025. Note that the Digital Omnibus agreement of May 2026 deferred the high risk deadlines, moving Annex III obligations to 2 December 2027 and Annex I to 2 August 2028. The compliance clock slipped. The threat clock did not.

What are the OWASP Top 10 risks for LLM applications in 2025?

LLM01 prompt injection, LLM02 sensitive information disclosure, LLM03 supply chain, LLM04 data and model poisoning, LLM05 improper output handling, LLM06 excessive agency, LLM07 system prompt leakage, LLM08 vector and embedding weaknesses, LLM09 misinformation and LLM10 unbounded consumption. If you run agents rather than a chat interface, pair it with the OWASP Top 10 for Agentic Applications, ASI01 to ASI10.

Testing the Testers: How to Red-Team Your Own AI Agents

Your agent reads a support ticket. The ticket contains a sentence written for the agent, not the customer. That is the whole attack.

There is no buffer overflow here. No malformed packet. Just plain English, placed where the agent will read it, phrased as an instruction. And because the agent has tools, that instruction becomes an action.

This is no longer theoretical. In March 2026 Palo Alto Networks Unit 42 published findings on web-based indirect prompt injection observed in the wild, cataloguing 22 distinct payload engineering techniques. The Cloud Security Alliance measured a 32 percent relative increase in malicious injection content between November 2025 and February 2026. Real payloads in that research forced $5,000 PayPal transfers, deleted backup folders and exfiltrated API keys.

IBM's 2026 breach study found that prompt injection breaches cost an average of $5.89 million, and that 92 percent of organisations that suffered an AI-related breach lacked proper AI access controls.

Adoption went up. Governance went down.

Darktrace surveyed 1,540 security leaders and practitioners across 14 countries. 78 percent deploy generative AI in at least one business function. 76 percent are worried about AI agent risk specifically.

And only 37 percent have a formal AI deployment policy. That figure fell 8 points year on year. Deployment accelerated and governance went backwards at the same time.

78%

Deploy generative AI in at least one business function (Darktrace 2026)

37%

Have a formal AI policy — down 8 points year on year

$5.89m

Average cost of a prompt injection breach (IBM 2026)

Gartner expects that by 2027, 40 percent of enterprises will demote or decommission autonomous AI agents because governance gaps were only identified after production incidents. Their analyst put the cause plainly: enterprises treat agent governance as binary, either locked down or fully trusted.

The agent does not need to be malicious. It just needs to be obedient to the wrong text.

Test against the Agentic Top 10, not the chatbot list

Most teams still test against the OWASP Top 10 for LLM Applications 2025, which runs LLM01 prompt injection through LLM10 unbounded consumption. It is a good list. It was written for applications where the model produces text.

If your system takes actions, you need the newer one. OWASP published the Top 10 for Agentic Applications on 9 December 2025, built with over 100 contributors and a review board including NIST, Microsoft, AWS and the Alan Turing Institute.

The ten agentic risks (ASI01–ASI10). Goal hijack, Tool misuse, Identity and privilege abuse, Agentic supply chain, Unexpected code execution, Memory and context poisoning, Insecure inter-agent communication, Cascading failures, Human-agent trust exploitation, Rogue agents. OWASP's own analysis found prompt injection maps to six of these ten categories. It is not one item on a list. It is the delivery mechanism for most of the list.

These attacks already have CVE numbers

EchoLeak, CVE-2025-32711, was a zero-click indirect prompt injection in Microsoft 365 Copilot that allowed information disclosure over a network. In April 2026 two more landed on the same day. CVE-2026-21520 in Microsoft Copilot exfiltrated customer data through injected SharePoint form input, and returned sensitive data even when safety mechanisms detected the attack. The same researchers showed Salesforce Agentforce treating a malicious lead capture form as a trusted instruction.

The Cloud Security Alliance counted eight major AI incidents in Q1 2026, and only one received a CVE. Prompt injection findings are often paid through bug bounties with no public advisory. There is no feed to subscribe to and no patch Tuesday to wait for. If you are not testing your own agents, nobody is doing it for you.

Regulators already asked for this

The EU AI Act, Article 55(1)(a), requires providers of general-purpose models with systemic risk to conduct and document adversarial testing. Those obligations applied from 2 August 2025. The Digital Omnibus agreement in May 2026 deferred the high-risk deadlines, moving Annex III to 2 December 2027 and Annex I to 2 August 2028. The compliance clock slipped. The attacker clock did not.

NIST AI 600-1, the Generative AI Profile, names red teaming repeatedly, including MP-5.1-005 on adversarial role-playing and chaos testing. MITRE ATLAS gives you the attacker-side vocabulary, and its 2026 restructure tags every technique by platform, including agentic AI.

What good looks like

OWASP's GenAI Red Teaming Guide splits the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis. Most teams do the first and skip the other three. The first is the one your model vendor already did.

Three practical rules follow. First, test behaviour, not just output. A wrong answer is a quality problem. A correct-sounding answer that triggers a transfer is a security problem.

Second, treat every agent as a privileged identity. Only 40 percent of organisations use access controls on AI models and data. Scope the tools, require confirmation for anything irreversible, and log every call.

Third, run it continuously. A new prompt, a new tool, a model upgrade, a new document in the knowledge base. Each is a change that invalidates the last test. Arxiis exists for exactly this: an autonomous red team that probes your agents and applications across 26 modules and 6 attack vectors, and re-earns a known-secure state every 24 hours.

You built something that acts on its own. Test it the way something else will.

The teams that get this right are not the ones that ban agents. They are the ones that treat an agent like any other high-privilege system: scoped, logged, attacked on purpose, and attacked again after every change.

Close your security gaps — continuously.

Arxiis is an autonomous AI penetration-testing platform that runs a coordinated crew of specialist agents through a full engagement in hours, not weeks — continuously, not annually. Every finding is CVSS-scored, MITRE ATT&CK-tagged, and auto-mapped to RBI CSF, CERT-In and nine more frameworks. It runs entirely on your own infrastructure. Nothing leaves the perimeter.

Testing the Testers:Red-Team Your Own AI Agents.