Darktrace's 2024 State of AI Cybersecurity report found that 92% of security professionals are concerned about AI agent risks to their organisations. The follow-up finding is the one that matters: only 37% of those organisations have a formal policy governing how AI agents operate within their environment. The gap between concern and governance is the space in which the risk actually lives.

AI agents — autonomous software systems that can plan, reason, and take actions within digital environments — have moved from research demonstrations to enterprise deployments in the past eighteen months. They exist in customer service workflows, code development pipelines, security operations centres, and data analysis functions. They have access to email systems, databases, APIs, and internal tools. Most of them were deployed by individual teams without going through a security review process.

The threat model for AI agents

The security risk from AI agents is not the science fiction scenario of a system going rogue. It is the mundane scenario of a system doing exactly what it was designed to do, in a context that its designers didn't anticipate, with access that was provisioned more broadly than it needed to be.

The primary attack surface is prompt injection. An AI agent that reads external content — emails, web pages, documents, API responses — can be manipulated by an attacker who crafts content designed to change the agent's behaviour. A customer service agent that can access order history, issue refunds, and update account details can be prompted by a malicious email to take actions that a human customer service representative would recognise as fraudulent but that the agent executes as a valid instruction.

92%
Of security professionals are concerned about AI agent risks (Darktrace 2024)
37%
Have a formal policy governing AI agent behaviour in their environment
63%
Are running AI agents in production with no formal governance framework in place

Broad access as a risk multiplier

The second major risk factor is access scope. AI agents are typically provisioned with the access their designers thought they would need, plus some margin for unanticipated use cases. In practice, this often means they have access to significantly more data and more functionality than they use in normal operation. When a prompt injection attack redirects the agent's behaviour, that broad access becomes the attack's blast radius.

A code development agent that has read access to all repositories in order to understand the codebase, write access to feature branches, and the ability to open pull requests — and that processes commit messages or issue descriptions as instructions — is a supply chain attack vector. An attacker who can place content in a repository that the agent reads can potentially direct it to exfiltrate code, introduce vulnerabilities, or create pull requests that appear to be legitimate development work.

The logging gap. 63% of organisations running AI agents in production have no formal governance framework. A significant portion of those also have no structured logging of agent actions that would allow incident responders to reconstruct what an agent did in the event of a security incident. Traditional incident response playbooks assume the actions in an environment were taken by humans or by known automated scripts. Agent actions are neither, and most SIEM deployments are not configured to capture, correlate, or alert on them.

What governance actually requires

A formal AI agent governance policy is not a document. It is a set of controls that change how agents are deployed, operated, and monitored. At minimum, it requires four things: a principle of least privilege applied to agent access (agents should have access only to the specific resources they need for their defined function, not broad access to the environment); input validation and sanitisation for any content the agent reads from external sources; structured action logging that allows reconstruction of what the agent did and why; and human-in-the-loop approval for high-impact actions such as financial transactions, data exports, and configuration changes.

These are not novel security controls. They are the same controls that would be applied to any privileged automated process. The gap is not that security teams don't know what controls to apply — it is that AI agents were deployed by non-security teams who didn't apply them, and the security team didn't have visibility into the deployment until after the fact.

The AI agent risk is not the agent going rogue. It is the agent doing exactly what it was built to do, with the wrong instructions.

Testing AI agent environments

The security testing requirement for environments with AI agents is different from traditional application security testing. Prompt injection is not a vulnerability class that conventional VAPT methodologies assess. Attack paths that run through agent behaviour — injecting a malicious prompt via an API response, redirecting an agent's file access to extract sensitive data, manipulating an agent's output to inject code into a downstream system — require testers who understand both the agent architecture and the attack techniques.

This is an area where the security testing industry has not yet caught up to the deployment reality. Most enterprises running AI agents in production have not had those agents tested for prompt injection, access control flaws, or supply chain attack vectors. The 92% concern figure in the Darktrace data reflects awareness of a risk that is largely unmitigated.

Close your security gaps — continuously.

Arxiis tests the full attack surface of your environment, including AI agent deployments, API integrations, and supply chain paths that conventional VAPT misses. 26 specialist modules, 6 attack vectors, 11 compliance frameworks. On-premise, in hours.