XBOW alternatives: a comparison that admits where XBOW is ahead

If you are comparing XBOW alternatives on headline solve rate, XBOW wins, and this post will not pretend otherwise. XBOW published the XBOW-104 benchmark and a public solve rate on it that is higher than ours. What follows is a comparison on the axes a buyer can actually verify, with the honest result on each.

First, what we will not claim

We do not claim to beat XBOW on XBOW-104. XBOW's public number is a full-104 figure. Our strongest number, ~82%, is on the level-1 tier only. Our cross-level result is 56% raw and 44% confirmed, run dated 19 October 2026. Those are not the same measurement, and a vendor who lines them up side by side as if they were is misleading you.

We also do not know XBOW's confirmed-versus-raw split or its measured false-positive rate, because we have not seen them published. That is not a criticism. It is a gap in what can be compared, and you should ask XBOW directly.

The axes that can be compared

Evidence per finding. Arxiis labels a finding proven only when its evidence binds to a record in a tamper-evident, hash-chained execution ledger. Otherwise the label is needs review. Ask any alternative, XBOW included, what the equivalent artefact is and whether you can inspect it per finding. This is the axis we would weight most heavily, and the one most demos skip.

Confirmed versus raw. We publish both. The 12-point gap between our raw and confirmed numbers is the set of results we could not bind to evidence. Ask each alternative for its gap. A vendor with no gap either confirmed everything, which is possible, or did not measure confirmation, which is more common.

Measured false positives. Arxiis: 0%, measured by an evidence audit of every confirmed solve on XBOW-104. The benchmark itself cannot measure false positives, since each challenge contains exactly one real bug, so any vendor's FP number needs a described method behind it.

Scope and approval control. Arxiis will not engage a non-local target without a written SCOPE + ROE authorization file. A scope guard checks every host. An operator approves each escalation. The watchdog stops the run at the time or token budget. These are hard checks, not prompt instructions. Ask each alternative which of these exist, and which are enforced outside the model.

Published weak spots. Ours: hard chained and blind attacks at 0–42% depending on sub-tier, and a cross-level confirmed rate roughly 30 points behind the best public claims. Ask each alternative for theirs.

Where XBOW is ahead

On breadth of public validation, XBOW is ahead. It published the benchmark the field uses, it has published CVE write-ups, and its headline solve rate on its own benchmark is higher than ours on any comparable measurement we have made. If your evaluation weights raw finding power on isolated targets above everything else, that matters and you should weigh it.

Where Arxiis is different

Arxiis is built for the team that acts on the report rather than re-tests it. Every proven finding has a receipt. The report is scrubbed of impact claims the ledger does not support. The system fails closed. Those choices cost us solve rate, visibly, and we think they are the right trade for authorized security teams who need to hand the report to someone.

The honest comparison matrix for XBOW alternatives has more "ask the vendor" cells than filled ones. Fill them yourself, with the nine questions in our evaluation scorecard, and weight them for your team. If the answer comes out XBOW, that is a fine answer.

A decision guide, not a ranking

Searches for "XBOW alternatives" usually come from one of four situations. Which one you are in changes the answer.

You want the highest published raw solve rate on isolated targets. XBOW's public number on its own benchmark is the one to look at first. Weigh it heavily and treat the other axes as tie-breakers.

You need findings your team can act on without re-checking. Evidence per finding, a confirmed rate and a measured false-positive rate matter most here. This is where Arxiis is designed to compete.

You need tight control over what a tool may touch. Look at how scope is enforced, who approves escalation, and what happens at the budget limit. Ask every vendor, XBOW included, whether those controls are checks outside the model or instructions inside it.

You are under audit or regulatory pressure. The artefacts behind each finding and the retention of evidence matter most. Ask what you will be able to hand an auditor after the test.

Most buyers are in more than one of these. List which two matter most to you before you read any vendor material.

Questions to put to every shortlisted vendor, XBOW included

Use the same list for everyone.

1. Show me the stored evidence behind one finding in a sample report.
2. What are your raw and confirmed solve rates, on which benchmark, tier and date?
3. How was your false-positive rate measured, and on what data?
4. What written document must exist before the tool engages a target?
5. Who approves escalation between hosts or tiers?
6. What does the report look like when the budget runs out mid-test?
7. Which vulnerability classes are you validated for, and which are you not?

A vendor who answers all seven in writing is easy to compare. One who answers four is telling you where their measurements stop.

Switching and running two tools in parallel

You do not have to pick one tool forever. Many teams run a pilot of two vendors on the same staging application under the same scope file, then compare what came back.

Compare three things: how many findings each vendor confirmed that your engineers agreed with, how many each vendor left unconfirmed and whether the labels were honest, and how much time your team spent separating real from unreal. Those three numbers are usually more informative than any headline score.

If you do switch, keep the evidence from the old tool. Records from earlier engagements are useful for audit and for checking whether previously reported issues were fixed.

Where this comparison could be wrong

We have written this from the outside. Everything we say about XBOW comes from its public material and the benchmark it published, and vendors change their products. If something above no longer matches what XBOW says today, the vendor's current page is the better source. We would rather be corrected than quote an out-of-date comparison.

What to put in the pilot report

If you run a pilot of two or more vendors, a short written report helps the decision survive scrutiny. A one-page structure works well.

Setup. The application tested, the scope file used, the dates, and the versions of each tool.

Results table. For each vendor: confirmed findings, confirmed findings your engineers agreed with, unconfirmed findings, and any findings that turned out to be wrong.

Behaviour notes. Whether the tool stayed in scope, how it handled the budget limit, and how clearly it labelled what it could not prove.

Effort. Hours your team spent on triage for each vendor.

Recommendation. One paragraph, with the weights you agreed before the pilot started.

Keeping the weights written beforehand matters. It stops the best demo from rewriting the criteria afterwards.

Reading a vendor's weak-spot answer

When you ask a vendor what the tool is bad at, listen for specifics. A useful answer names a class of vulnerability, a tier of difficulty or an environment type, and gives a number or a plan.

A less useful answer is "we are constantly improving" or "there are no meaningful gaps." Every tool has a boundary, and a vendor who has measured will know where it is.

Ours is hard chained and blind attacks, where results on XBOW-104 range from 0% to 42% depending on sub-tier. We state it because you will find it out anyway, and it is better to hear it up front.

When another vendor is the right answer

Sometimes the honest outcome of a comparison is that Arxiis is not the right fit. If your priority is raw finding power on isolated targets above everything else, or your estate is mostly internal infrastructure, another tool may serve you better.

We would rather you chose the right tool than the wrong one from us. That is also why this post names where XBOW is ahead.

Frequently asked questions

Does Arxiis beat XBOW on the XBOW-104 benchmark?

No. XBOW's public number is a full-104 figure and is higher than Arxiis's cross-level result of 56% raw and 44% confirmed. Arxiis's ~82% applies to the level-1 tier only. The measurements are not equivalent and should not be compared as if they were.

How should I compare XBOW alternatives?

On evidence per finding, confirmed versus raw solve rate, measured false-positive rate and its method, scope and approval controls enforced outside the model, and each vendor's published weak spots.

Which questions should I ask every AI pentesting vendor?

Show the stored evidence behind one finding; the raw and confirmed solve rates with benchmark, tier and date; how the false-positive rate was measured; what document must exist before engaging a target; who approves escalation; what the report looks like when the budget runs out; and which vulnerability classes the tool is and is not validated for.

Can I run two AI pentesting tools side by side?

Yes. Run both on the same staging application under the same scope file, then compare how many confirmed findings your engineers agreed with, how many were left unconfirmed and whether the labels were honest, and how much time your team spent separating real from unreal.