Autonomous pentesting vs manual pentesting: an honest split of the work

Autonomous pentesting is better than a human tester at some parts of a web-application test and clearly worse at others. The useful question is which parts you hand over. We build an autonomous system, Arxiis, and we would not run it without a senior tester in the loop. This post names where each side wins, using our own numbers and our own weak spots.

Two positions worth taking seriously

The first position says AI will replace manual testing. Models now find real bugs in real applications. They do not get tired, they do not skip the dull checks, and they work at any hour. On this view, senior testers are a transitional cost.

The second position says autonomous testing is a scanner with better marketing. Manual testers understand the application, the business and the attacker. A tool that cannot read a requirements document and spot the flaw in a refund flow is not pentesting.

Both positions are half right. Below is where we think the line sits today, for authorized testing of web applications and APIs. We have not measured anything else, so we say nothing about it.

What autonomous testing does better

Repeatability. A manual test is shaped by who ran it and how many days they had. An Arxiis run starts from the same written SCOPE + ROE authorization file each time. The scope guard checks every request against that file, and the crew writes every command to the same execution ledger. The model's individual choices will differ from run to run. The boundaries, the checks and the record format do not. You can run the same scope next week and compare like with like. A consultant's second visit is a different engagement.

Coverage of common web-bug classes. The level-1 tier of XBOW-104 is made of common web-application bugs. Arxiis solves about 82% of it on XBOW-104 (19 October 2026), level-1 tier only. That is the routine sweep of SQLi, SSTI, SSRF, XXE, JWT and LFI classes across every in-scope endpoint. It is also the work senior testers most often run short of time for. A crew of operator-agents does not lose attention on the fortieth endpoint.

Evidence logging. Manual testers keep notes, screenshots and proxy history, and the quality varies by person and by deadline. Our crew writes every command to a tamper-evident, hash-chained execution ledger. A finding is labelled proven only when its evidence binds to a real ledger record. Everything else is labelled needs review. Report language is then deterministically scrubbed of claims the evidence does not support. Our measured false-positive rate across XBOW-104 sweeps is 0%, with every confirmed solve independently re-observed. The record has the same shape on every run, which is what an auditor asks for.

Stopping honestly. When the budget or time window runs out, the watchdog stops the run. It writes a partial report that says it is partial and lists what was not reached. Any test that runs out of time, human or automated, can end with a report that reads as complete. Ours is built not to.

What senior human testers do better

Business logic. The OWASP Web Security Testing Guide says business-logic flaws cannot be found by a vulnerability scanner and depend on the tester's skill and creativity. An LLM agent is not a scanner, and it can read more context than one. But we have not measured Arxiis on business-logic flaws, so we make no claim there. A tester who knows a discount should apply once, or that a manager should not approve their own expenses, is testing intent. That knowledge lives in people and documents, not in HTTP traffic.

Chained and blind attacks. This is our known weak spot. On the hard chained and blind tiers of XBOW-104, Arxiis solves 0–42% depending on sub-tier, a known weak spot. These are problems where one finding matters only because of a second, or where the application gives no direct feedback. They need a hypothesis held across many steps. Senior testers are better at that today. Across all levels, Arxiis solves 56% raw and 44% confirmed, roughly 30 points below the best publicly claimed systems.

Judgment about what matters. Every finding has a technical severity and a business severity. A human knows an internal hostname leaking from a marketing site is not the same as one leaking from the payments service. Our labels describe evidence, not business impact, and we make no breach-impact claims. Ranking findings for a specific business is a human call.

Scoping and rules of engagement. Deciding what to test, what to leave alone and when testing may run is an agreement between people with authority. Arxiis will not engage a non-local target without a written authorization file. The system cannot write that file or widen it. Someone has to own it, and that person should know the estate.

Noticing what feels wrong. Testers notice when a staging host looks like production, or a system seems fragile. Our scope guard blocks hosts outside the file, and the approval gate holds escalation until a human says yes. Those are hard stops, not intuition. The intuition stays with the operator.

The split, in one table

Leverage, not replacement

Our position is that an autonomous crew is leverage for senior testers, not a replacement for them. Picture one senior person supervising a crew. The crew takes the repeatable sweep and the record-keeping. The senior person writes the scope, approves escalation, and works the needs review queue. Then they spend the hours they saved on the logic flaws and chains the crew cannot reach.

That changes a senior tester's week. Less time on the routine sweep. More time on the work clients pay senior rates for. And a cleaner handoff, because every finding the crew reports either carries its ledger record or says it needs a human.

It is also why Arxiis ships an operator console, a headless CLI and an API, not a start-and-walk-away button. The model never approves its own escalation and never upgrades its own finding. Supervision is part of the design, not a setting.

The concession

There is a version of the replacement argument that holds. A small application with a simple flow may only ever get an authorized autonomous run plus a human review of the results. That can be better than no test. Teams without a senior tester will use tools this way, whatever vendors say.

The line will also move. Our chained and blind results are a weak spot we are working on, not a permanent ceiling. When the numbers change, we will report them with the benchmark name, the tier and the run date, as we do now.

What will not move is who owns the decisions. Scope, escalation and the final word on an unproven finding stay with a person. Those are decisions about trust, not technique.

How to decide for your own programme

Start from the application, not the tool.

If most of the risk sits in common web-bug classes across many endpoints that change often, let supervised autonomous runs carry the routine sweep. Keep a senior tester for review, approvals and the hard cases.

If the risk sits in workflows, money movement or permissions, let the senior tester lead. Use autonomous runs to clear the routine classes so the tester does not have to.

If an auditor or a customer contract expects a human-led test, keep it. Use supervised autonomous runs between those tests, not instead of them.

In every case, ask both sides for the same three things. What was tested. Under which written authorization. What evidence sits behind each finding.

Questions to settle before you shift work

Who owns the needs review queue? An unproven finding is a work item, not a lower severity. Name the person who will close it, and agree how quickly.

Which classes is the tool measured on, at which tier? A level-1 number says little about chained attacks. Ask for the benchmark, the tier and the run date for every figure.

What will the senior tester do with the saved hours? Write it down before the first run. If the answer is "nothing in particular", the hours tend to disappear.

What does the tool do when it is unsure, out of scope or over budget? You want it to stop and say so. A tool that keeps going and reports is moving judgment away from your team.

Frequently asked questions

Is autonomous pentesting better than manual pentesting?

Neither is better across the board. Supervised autonomous testing is better at repeatable runs against one written scope. It is also better at sweeping common web-bug classes across many endpoints and logging evidence for every finding. Senior human testers are better at business logic, chained and blind attacks, judgment about business severity, and scoping.

Will AI replace human penetration testers?

Not in our design. Arxiis is leverage for senior testers, not a replacement. On XBOW-104 it solves 0 to 42% of hard chained and blind attacks, depending on sub-tier, a known weak spot. A human must own scope, approve escalation and review unproven findings.

What does Arxiis do better than a manual tester?

It re-runs the same authorized scope with the same checks and record format. It sweeps common web-bug classes, solving about 82% on XBOW-104, level-1 tier only. It writes every command to a tamper-evident, hash-chained ledger. A finding is labelled proven only when its evidence binds to a ledger record; otherwise it is labelled needs review.

Can autonomous pentesting find business logic flaws?

We have not measured Arxiis on business-logic flaws, so we make no claim. The OWASP Web Security Testing Guide says these flaws cannot be found by a vulnerability scanner and depend on the tester's skill and creativity. Treat business logic as senior-tester work.

How should a senior tester spend time when using an autonomous tool?

On the work the crew cannot own. That means writing the scope and rules of engagement, and approving or declining escalation requests. It also means closing the needs review queue and working business-logic flaws and chained attacks.