How to evaluate AI pentesting tools: the 9-point scorecard we'd want used on us

Every AI pentesting demo shows the same three things: a dashboard, a findings list, and one large solve-rate number. None of them tells you whether the tool's output can be trusted without re-testing it. The nine questions below do. We score Arxiis against each one at the end, including the two we score badly on.

1. Can it show the evidence behind one finding?

Pick any finding in the demo report. Ask for the exact request that was sent, the exact response that came back, and where that record lives.

A good answer is a record you can inspect: timestamped, with the raw request and response, in a store the model cannot edit. A weak answer is a summary paragraph, a screenshot, or "the AI verified it".

At Arxiis the record is an entry in a tamper-evident, hash-chained execution ledger. A finding is labelled proven only when its evidence binds to that record. Ask any vendor for their equivalent.

2. Does it distinguish raw from confirmed?

Ask for two numbers, not one. Raw: the tool got the flag or triggered the bug. Confirmed: the result was independently re-observed and the finding is backed by evidence.

Any vendor who has run a benchmark seriously has both numbers. A vendor who only has one either did not measure confirmation or is choosing not to show it.

Arxiis on XBOW-104, run dated 19 October 2026: 56% raw, 44% confirmed.

3. Is the false-positive rate measured, and on what?

"Low false positives" is a claim. "0% measured false positives on XBOW-104, from an evidence audit of every confirmed solve" is a measurement. Ask for the benchmark, the method, and the date.

Note what the measurement covers. Most public benchmarks contain exactly one real bug per challenge, so they cannot measure false positives directly. A separate audit is needed, and the vendor should be able to describe it.

4. Which benchmark, which tier?

A solve rate without a tier is not a number. ~82% on level-1 common web bugs and 44% cross-level can both be true of the same system, and they mean different things. Ask for the breakdown by difficulty, and ask specifically about the hardest tier. That is where the honest vendors sound worst, and where the gap between tools is widest.

5. What happens when it is not sure?

Run a target with a bug the tool cannot fully confirm, a blind SSRF with egress blocked, for instance. Then look at the report.

A fail-open tool reports the finding anyway, often with a severity. A fail-closed tool reports it as unconfirmed, with lower confidence or a separate label, and says why. The second behaviour is the one you want on a production estate, and it will cost that tool points on a benchmark. Decide which you are paying for.

6. What does it refuse to touch?

Give it a target with no authorization on file. It should not start. Give it an authorized scope and a reachable host outside it. It should not cross the line, and the attempt should be logged.

Ask what the authorization artefact is. At Arxiis it is a written SCOPE + ROE file, and the crew will not engage a non-local target without one. Ask whether the scope guard is a prompt instruction or a hard check. A prompt can be argued out of; a check cannot.

7. Who approves escalation?

When the tool finds a foothold and wants to go further, who says yes? If the answer is "the agent decides", you are evaluating a tool that can approve its own scope creep. If the answer is "an operator approves each escalation", you are evaluating a tool with a human in the loop. Ask to see the approval step in the interface.

8. What stops it running forever?

Ask about the budget. Time, tokens, requests per host. Then ask what the report looks like when the budget is hit mid-engagement. A good answer is a partial report with every finding honestly labelled. A bad answer is "it finishes the job".

9. What can it not do, in writing?

Ask for the list. Vulnerability classes it does not cover. Environments where it has not been tested. Tiers where it scores poorly. A vendor who answers "nothing, it's comprehensive" has failed the question.

How Arxiis scores

Two weak rows, in writing. We would rather you see them here than find them in month two.

The question we cannot answer for you is how to weigh row 4 against rows 3 and 5. A tool that scores higher on hard chains by accepting some false positives might be the right choice for a red team that re-verifies everything anyway. For a team that acts on the report directly, we think evidence wins. Score it for your own team.

A scoring sheet you can copy

Give each of the nine questions a score from 0 to 3, then weight the rows for your team. A simple sheet looks like this.

Weights depend on how your team uses the output. A team that acts directly on the report should weight evidence per finding, measured false positives and behaviour when unsure most heavily. A team that re-verifies everything anyway might weight coverage and benchmark tiers more.

Write the weights down before you see any vendor's answers. It stops a strong demo from moving the goalposts.

Common answers, and what they usually mean

"Our AI verifies every finding." Ask what does the verifying. If it is the same model that produced the finding, the verification adds little.

"We have a very low false-positive rate." Ask for the number, the method and the dataset. If any of the three is missing, treat it as a claim rather than a measurement.

"We cover the full OWASP Top 10." Coverage of a category is not the same as validated performance. Ask for the results per class.

"It is fully autonomous." Ask what stops it from approving its own next step, and who is accountable when it acts. For authorized testing, human approval is usually a feature, not a gap.

"Benchmarks don't reflect real environments." Often true, and a fair point. It is not a reason to have no measurements at all. Ask what they measure instead.

Running the scorecard in a pilot

Scoring answers on paper is the first half. The second half is checking them against behaviour during a short pilot.

Choose one staging application and write one scope file. Give every shortlisted vendor the same file. Then check the tool's behaviour against what it said: whether each confirmed finding came with stored evidence, whether unconfirmed findings were labelled honestly, whether the tool stayed inside the scope, and whether the run stopped cleanly at the budget.

Any gap between what a vendor said and what the pilot showed is worth recording, because it is the most reliable signal you will get.

Who should be in the room

An AI pentesting evaluation works best with three roles. A security engineer who will use the output daily. A security leader who owns the risk decision. And someone from procurement or legal who cares about the authorization and contract terms. Each of the nine questions maps to at least one of them, and the scorecard works best when all three score independently before comparing notes.

When to re-run the evaluation

Tools change, and so do your needs. Repeat the scorecard after a major version change, when your scope grows, or once a year at renewal. Keep the old sheets so you can see whether a vendor's answers got stronger or weaker over time.

Frequently asked questions

What is the most important question when evaluating an AI pentesting tool?

Whether it can show the evidence behind a single finding: the exact request, response, and the record they are stored in. If it cannot, every other number depends on trusting the model.

Why should I ask for raw and confirmed solve rates separately?

Raw counts any flag captured. Confirmed counts only results that were re-observed and backed by evidence. The gap shows how much of the headline number you would have to re-verify yourself. Arxiis reports 56% raw and 44% confirmed on XBOW-104.

How do I score AI pentesting vendors fairly?

Score each of the nine questions from 0 to 3, where 3 means a number, method, date and a way to check it. Write your weights down before seeing any answers, and check the answers against behaviour in a short pilot under one shared scope file.

Who should take part in an AI pentesting evaluation?

A security engineer who will use the output, a security leader who owns the risk decision, and someone from procurement or legal who cares about authorization and contract terms. Each scores independently before comparing.