AI pentesting vendors all demo the same. Here is what actually separates them.

The AI pentesting market is mostly indistinguishable from the demo, and that is the buyer's problem to solve, not the vendor's. Every vendor, us included, shows a dashboard, a findings list, and a big number. The differences that matter show up only when you ask questions the demo was not built to answer.

The case for judging by the demo

There is a fair argument for it. A demo on a target you chose shows real behaviour: what it found, how it wrote it up, how long it took. If a tool finds the bugs you planted and misses nothing obvious, that is evidence. Benchmarks are abstract; your own target is not. And the vendor's slide about architecture is marketing until you see the output.

That argument is right about one thing. The output matters more than the slide. It is wrong about what the output shows.

Why the demo cannot separate vendors

A demo target has a small number of real bugs that the vendor knows about. Any tool built in the last two years will find most of them. What the demo cannot show is the three properties that decide whether you can act on the report without re-testing it.

Whether a finding carries its own evidence. On a demo, you already know which findings are real, so you never need to check. On a production estate with two hundred findings, you do. The question is whether each finding points at a record of the exact request and response, stored where the model cannot edit it, or whether it points at a paragraph the model wrote.

Whether the headline number is raw or confirmed. Vendors who measure both have both. Arxiis, on XBOW-104 dated 19 October 2026, solves 56% raw and 44% confirmed, and the 12-point gap is the set of results we could not bind to evidence. A vendor showing one number is either not measuring confirmation or not showing it. Either way you are looking at the raw figure.

What it does when it is unsure, out of scope, or over budget. The demo never hits any of those. Production hits all three in the first hour. Fail-open tools keep going and report. Fail-closed tools stop and label. Arxiis will not engage a non-local target without a written SCOPE + ROE file, an operator approves each escalation, and the watchdog stops the run when the budget is gone. You only learn which kind of tool you have by asking, or by finding out later.

The fourth difference is honesty about weak spots, and it is the easiest to test. Ask every vendor what they are bad at. Ours: hard chained and blind attacks, 0–42% depending on sub-tier, and roughly 30 points behind the best public cross-level claims. A vendor with no answer to that question has given you one.

The concession

Some buyers do not need any of this. A mature red team that re-verifies every finding by hand gets the same value from a tool with a 5% false-positive rate and a higher solve rate as from one with 0% and a lower one. If you have that team, weigh raw coverage more heavily than we would, and discount the evidence argument. The architecture we chose is built for teams that act on the report, not teams that audit it.

What separates them

Not the dashboard. Not the list. Not the number. Whether a finding can show its receipt, whether the number is confirmed, whether the system stops when it should, and whether the vendor will tell you where it fails. Four questions, none of which the demo can answer, all of which take ten minutes to ask.

What to put in front of procurement

Procurement teams compare vendors on price, terms and a feature grid. None of those three captures the differences above, so the security team has to translate them into language a contract can hold. Four clauses do most of the work.

An evidence clause. "For each finding reported as confirmed, the vendor will provide the underlying request, response and record identifier, stored in a form the vendor's model cannot alter." This turns a sales claim into a deliverable you can check at the first report.

A measurement clause. "The vendor will state its solve rate on a named public benchmark as two figures, raw and confirmed, with the benchmark tier and run date, and its measured false-positive rate with the method used." A vendor who will not sign this has told you which number they have been showing.

A control clause. "The vendor's system will not engage any host not listed in the written authorization file, and will not escalate between hosts or tiers without the approval of a named person." This is the clause your legal team will care about most, because it is what keeps an authorized test authorized.

A limits clause. "The vendor will list, in writing, the vulnerability classes and environments its system is not validated for." It sounds unusual. It is the cheapest insurance in the contract.

Arxiis signs all four as written. We would rather lose a deal on a clause than win one on a claim we cannot back.

The hidden cost in the comparison

Most vendor comparisons count licence cost and stop. The larger cost often sits after the report arrives: the hours your team spends deciding which findings are real.

Here is a simple way to estimate it for your own team. Take the number of findings a typical report contains. Multiply by the share you would need to re-check by hand. Multiply again by the time one re-check takes your engineers. The figures are yours, not ours, and they vary widely between teams. What matters is that the number is usually bigger than people expect, and that it is the number an evidence-bound report is designed to shrink.

If a tool labels every finding it cannot prove, you re-check only that labelled set. If it does not, you re-check everything, because you cannot tell the proven findings from the guesses.

Reporting the choice to your board

Boards rarely want the technical comparison. They want to know three things: what you bought, what it covers, and what it does not.

A good one-page summary for an AI pentesting purchase answers each in a sentence. What we bought: a human-supervised testing platform whose confirmed findings each carry a stored record of evidence. What it covers: web applications and APIs inside a written scope. What it does not cover: internal networks, social engineering, and anything outside the scope file. Then one line on the honest weak spot, because the board will hear about it eventually and it is better that they hear it from you.

A short glossary for the buying committee

Raw solve rate. The share of benchmark challenges where the tool reached the goal at least once.

Confirmed solve rate. The share where the result was also backed by stored evidence and independently re-observed.

Measured false-positive rate. The share of reported confirmed findings that turned out not to be real, with the method used to check.

Fail-closed. A system that stops and reports when it is unsure, out of scope or over budget, rather than continuing.

Authorization file. The written document that names what may be tested, when, and by whom. At Arxiis it is the SCOPE + ROE file.

One last test before you sign

Ask the vendor to put its answers to the four questions in an email. Written answers are easier to compare, easier to forward to legal, and easier to hold the vendor to at renewal. If the written version is weaker than the sales call, trust the written one.

Frequently asked questions

How do I compare AI pentesting vendors?

Ask four things the demo cannot show: whether each finding points to a stored record of the request and response, whether the solve rate is raw or confirmed, what the tool does when unsure, out of scope or over budget, and what the vendor says it is bad at.

What are Arxiis's weak spots?

Hard chained and blind attacks, where it scores 0 to 42% on XBOW-104 depending on sub-tier, and a cross-level confirmed rate of 44% (56% raw) that is roughly 30 points behind the best publicly claimed systems.

What contract clauses should I use when buying an AI pentesting tool?

Four: an evidence clause (each confirmed finding comes with its stored request, response and record identifier), a measurement clause (raw and confirmed solve rates with benchmark, tier and date, plus a measured false-positive rate), a control clause (no host outside the written authorization file, no escalation without a named approver), and a limits clause (classes and environments the tool is not validated for, in writing).

How do I estimate the re-verification cost of an AI pentesting tool?

Multiply the number of findings in a typical report by the share your team would need to re-check by hand, then by the time one re-check takes. Use your own figures. A tool that labels every unproven finding lets you re-check only that labelled set instead of the whole report.