Human-supervised isn't a compromise. It's the design.

Most AI pentesting vendors are racing toward full autonomy. Arxiis is not, and that is a design decision rather than a missing feature. In our system a human operator owns three decisions the model is never allowed to make: what is in scope, whether to escalate, and what happens to a finding the evidence does not support. This post is the case for keeping those three decisions human.

The case for full autonomy

It deserves a fair hearing. Approvals are slow, and slow means less coverage per hour. Human reviewers are inconsistent, and a tired operator waves things through. The whole point of automating a penetration test is to remove the bottleneck, and the bottleneck is people. If the model is good enough to find the bug, the argument goes, it is good enough to decide the next step.

For some kinds of automation that argument wins. Code review, for instance: a wrong decision produces a bug that a test will catch later. In an authorized security test the decisions are different in kind. They are about where a tool is permitted to go on someone else's systems, and a wrong answer there is not something a later test catches. It is a breach of the authorization the test was given.

What the operator controls

Three things, and only three. Everything else is automated.

Scope. The operator loads the written SCOPE + ROE authorization file. The scope guard reads it and checks every outbound request against it. The model cannot widen scope, cannot add a host, and cannot argue the guard out of a block. If a request targets a host outside the file, the run halts and reports. This is a deterministic check, not a prompt instruction.

Escalation. When the crew finds a foothold and wants to go further, moving from one tier to another, or doing anything on a host listed as enumeration-only, it raises an approval request in the operator console. The run waits. A human reads what the crew wants to do and why, and approves or declines. The decision and the person who made it are written to the execution ledger. The system cannot approve its own next step, by construction.

Review. A finding is labelled proven only when its evidence binds to a ledger record. Everything else is needs review, and that label is a work queue for the operator, not a lower severity. The operator decides whether a needs review finding is worth a manual look, gets dropped, or gets re-run with a different confirmation strategy. The model never upgrades its own finding.

Why these three

They are the three decisions where a mistake cannot be undone by the next step of the same system. A wrong exploit attempt can be retried. A wrong scope decision has already touched something it should not have. A wrong escalation has already gone somewhere the authorization did not cover. A wrong proven label has already told a client something false. Each of those is a decision about trust rather than about technique, and trust is the thing a human has to carry.

There is a second reason. Every one of those decisions is recorded in the ledger with the operator's identity. When a client asks why the test did something, the answer is a named person and a timestamp, not a model's reasoning trace.

The concession

Full autonomy would let Arxiis run without an operator present. Overnight runs, continuous runs, runs across a hundred targets at once with nobody watching. That is real capability, and some buyers want it. We have deliberately not built it. For a team with the staff to supervise, that costs nothing. For a team that wanted to switch the tool on and walk away, Arxiis is the wrong tool, and we would rather say so here than in month two.

It also costs us speed. A run that waits for approval is slower than one that does not. On a benchmark with no scope boundaries and nothing to escalate into, that cost is invisible. On a production estate it is the point.

The position

Human-supervised is not autonomy with a safety rail bolted on afterwards. It is the architecture: a crew of operator-agents that does the finding, bolted to a human who does the deciding. The pitch is leverage for the senior people you already have, not their replacement. One operator supervising a crew covers more ground than one tester working alone, and every decision that matters still has a name on it.

What a supervised engagement looks like day to day

"Human-supervised" can sound abstract, so here is how it works in practice for the person doing the supervising.

An operator starts the engagement by loading the signed authorization file and confirming the scope and window. During the run, the crew works through the in-scope hosts on its own: mapping them, probing, and recording each step to the ledger. The operator watches progress in the console and does not need to direct each step.

When the crew reaches a decision point, such as a request to move from one tier to another, the run pauses and an approval request appears. The operator reads what is being asked and why, checks it against the scope file, and approves or declines. At the end, the operator reviews the findings labelled needs review and decides what to do with each.

So the operator's time goes on the three decisions that matter, not on watching the tool work.

What makes a good operator

The role does not require a team of senior red teamers, but it does need someone with security judgement. In practice, three qualities matter.

Understanding of the target. An operator who knows the application can tell quickly whether an escalation request makes sense.

Comfort saying no. The approval gate is only useful if declining is a normal outcome. Operators should feel free to stop a run that looks wrong.

Care with evidence. Reviewing needs review findings means reading the stored request and response and deciding what they show.

Many teams give the role to an experienced security engineer, with a senior tester available for hard calls.

Setting approval service levels

Approvals that sit unanswered stall the run. Agree a response time up front. A common pattern is a short target during the testing window, say a few minutes during business hours, with a named backup for when the primary operator is away. Record both in the authorization file.

It is also worth deciding what happens if nobody answers. The safe default is that the run waits, and eventually halts and reports, rather than carrying on.

What to put in your internal policy

If your organisation uses AI-assisted testing, a short internal policy keeps it consistent. Cover who may start an engagement, who may approve escalations, what the written authorization must contain, how evidence is stored and for how long, and who reviews the findings.

Keep it to a page. A policy people read beats a long one they do not.

Where full autonomy would fit, and where it would not

Some work suits unsupervised automation: running the same safe checks across many applications on a schedule, for example. Authorized testing of a specific scope, where a wrong decision means touching something outside the permission given, is different. Arxiis is built for the second case, which is why the human decisions are part of the design rather than an option.

Measuring whether supervision is working

If you run supervised engagements regularly, a few simple measures show whether the model is working for your team.

Track how long approval requests wait before an answer, and how often they are declined. A request that is never declined may mean the operator is approving without reading, and one that always waits too long may mean the response plan needs a backup. Track how many needs review findings are resolved, and how long they sit in the queue.

Review these figures every quarter. They tell you whether the human decisions are adding the control you built them for, or have turned into a formality.

Training a new operator

New operators learn fastest by shadowing a few engagements before running one alone. A useful first week covers how to read the authorization file, what each console view shows, how to judge an approval request, and how to read a ledger record.

Give them a short checklist for each approval: Is the target in the scope file? Does the request match the rules? Is the reason clear? If any answer is no, decline and ask.

Frequently asked questions

What does the human operator control in Arxiis?

Three decisions: scope, via the written SCOPE + ROE authorization file that the scope guard enforces; escalation, which the crew must request and a human must approve from the operator console; and review of findings labelled needs review. The model cannot make any of the three.

Can Arxiis run fully autonomously without an operator?

No. Unsupervised operation is capability Arxiis has deliberately not built. A human operator must be present to approve escalation and review findings.

How much of an operator's time does a supervised AI pentest take?

The crew does the finding on its own. The operator loads the authorization file, answers approval requests when the run pauses at a decision point, and reviews findings labelled needs review at the end, so time goes on the decisions that matter rather than on watching the tool.

What happens if no one answers an approval request?

The safe default is that the run waits and then halts and reports rather than carrying on. Agree a response time and a named backup operator in the authorization file.