What 44% confirmed on XBOW-104 actually means, and why we published it

Arxiis solves 44% of XBOW-104 confirmed, 56% raw, on a cross-level run dated 19 October 2026. We could have led with 56. We lead with 44, and this post explains the gap, the method, and the parts of the benchmark we are still bad at.

What XBOW-104 is

XBOW-104 is a set of 104 web-application challenges published by XBOW. Each challenge is a self-contained target with a known vulnerability and a flag. Capture the flag, and the challenge counts as solved.

The set is stratified by difficulty. Level-1 challenges are single, common web bugs: reflected XSS, straightforward SQLi, path traversal. Higher levels chain steps, hide the signal, or require blind techniques where the target gives no direct feedback.

That stratification is why we use it. A flat list of 104 wins or losses hides where a system is strong and where it fails. A level-by-level breakdown does not.

How we ran it

Every challenge ran under the same conditions:

• One engagement per challenge, with a written SCOPE + ROE authorization file naming the single target host.
• The flagship configuration, validated on Claude Sonnet. Arxiis is model-agnostic, but these numbers are for that configuration only.
• A fixed time and token budget per challenge, enforced by the watchdog. When the budget ran out, the run stopped and reported what it had.
• Every command written to the tamper-evident, hash-chained execution ledger.

No challenge was retried after a failure. No prompt was tuned against a specific challenge.

"Raw" versus "confirmed"

This is the part that matters.

A raw solve means the agent produced the correct flag at some point during the run.

A confirmed solve means the flag was produced, and the finding that led to it is labelled proven, and an independent re-observation reproduced the result. proven is not a judgement call. A finding gets that label only when its evidence binds to a real ledger record. Otherwise the label is needs review.

The 12-point gap between 56% and 44% is the set of challenges where the agent got the flag but could not bind the path to evidence we would put our name on. Some of those were lucky guesses. Some were correct exploitation with a broken evidence chain, usually a missed out-of-band callback. We count none of them.

If a vendor quotes you a single solve rate, ask which of these two it is.

Results by tier

Benchmark: XBOW-104. Run date: 19 October 2026. Configuration: flagship on Claude Sonnet.

The weak spot

On the hard chained and blind challenges, Arxiis scores between 0% and 42% depending on the sub-tier. That is a known weak spot, and it is where most of the gap to the best publicly claimed systems sits.

Two failure modes account for most of it. First, multi-step chains where the signal from step one is faint and the agent commits to the wrong branch. Second, blind techniques where our signal-confirmation engine needs an out-of-band callback that the challenge environment does not allow. In the second case the agent sometimes has the right payload and cannot prove it, so the finding stays needs review and the solve stays unconfirmed. That is the system working as designed, and it costs us points.

What the number does not tell you

XBOW-104 measures whether a system can find and exploit a known bug in an isolated target. It does not measure false positives, because every challenge has exactly one real bug. It does not measure scope discipline, because there is nothing out of scope to wander into. It does not measure what happens on a messy production estate with rate limits, WAFs, and twelve interconnected services.

Our 0% measured false-positive rate comes from a separate evidence audit of the confirmed solves, not from the benchmark itself. Read it that way.

Where we are against public claims

Arxiis is roughly 30 points below the best publicly claimed results on cross-level benchmarks, measured against results we consider methodologically comparable. Some systems now self-report 104/104. We have not seen a published protocol for those runs that distinguishes raw from confirmed, so we do not compare against them directly.

What we are doing about it

Three things, in order. A second-signal strategy for chained challenges, so the agent holds two candidate branches instead of committing early. A differential-response confirmer for environments that block out-of-band callbacks. And wiring the already-built sqlmap and commix adapters into the runtime, which is not done yet and which we will not claim until it is validated.

The open question is whether closing the gap on hard chains is worth a higher false-positive rate. Right now we are choosing zero false positives over a bigger headline number. If that trade ever flips, we will publish the new numbers the same way we published these.

How to read any AI pentesting benchmark claim

A benchmark number is only useful once you know what sits behind it. Five questions turn a headline figure into something you can compare.

Which benchmark, which tier? A result on the easiest tier and a result across all tiers are different measurements, even when the percentages look close. Ask for both, and ask which one the vendor is quoting.

Raw or confirmed? Raw means the tool reached the goal at least once. Confirmed means the result was also backed by stored evidence and re-observed independently. Our own figures show why the split matters: 56% raw and 44% confirmed on the same run.

One attempt or best-of-several? A tool allowed ten attempts per challenge will score higher than one allowed a single attempt. Ask how many attempts each challenge received, and whether any challenge was retried after a failure.

Was anything tuned against the benchmark? Prompts, tool settings or strategies adjusted for specific challenges inflate a score without improving the tool. Ask whether the run used the same configuration the customer receives.

What was the budget? Time, tokens and request limits per challenge change results. A run with no budget cap can look strong and still be impractical to buy.

A vendor who answers all five in writing has measured carefully. A vendor who cannot has told you something useful too.

What the benchmark does not tell you about your estate

XBOW-104 is a controlled set of isolated challenges. Your estate is not. Three differences matter when you translate a benchmark result into a purchase decision.

Your applications are bigger. A challenge contains one real issue in a small application. Yours contain hundreds of routes, shared libraries and years of accumulated behaviour.

Your environment pushes back. Rate limits, web application firewalls, session timeouts and flaky staging environments all change how a tool behaves. None of that is present in a clean benchmark target.

Your scope has edges. A benchmark target has nothing out of scope to wander into. Your real scope file does. Whether a tool respects it is a separate question from whether it can solve challenges.

So treat a benchmark as evidence about one thing, the tool's ability to find and prove a known issue in an isolated target, and use a pilot to learn about the rest.

A short pilot that answers the rest

You do not need a long evaluation. A two-week pilot on one staging application, under a written scope file, tells you more than any slide.

During the pilot, record four things. How many findings the tool labelled confirmed, and how many of those your engineers agreed with on review. How many findings it left unconfirmed, and whether the labels were honest. How long your team spent deciding what was real. And whether the tool stayed inside the scope file the whole time.

If the confirmed set holds up on review, the unconfirmed set is clearly labelled, and the tool never left scope, you have the evidence that matters. The benchmark number then becomes context rather than the deciding factor.

How we plan to keep this post honest

Numbers on this page carry a run date, the configuration used and the tier. When we re-run the benchmark, we will publish the new results next to these and keep the old ones visible, so you can see the direction as well as the level. If a future run scores lower on any tier, it will appear here too.

Frequently asked questions

What is the difference between a raw and a confirmed solve on XBOW-104?

A raw solve means the agent produced the correct flag. A confirmed solve means the flag was produced, the finding is labelled proven because its evidence binds to a record in the execution ledger, and an independent re-observation reproduced it. Arxiis scores 56% raw and 44% confirmed.

What is Arxiis's false-positive rate on XBOW-104?

0% measured, from an evidence audit of every confirmed solve. The benchmark itself does not measure false positives, because each challenge contains exactly one real bug.

Where does Arxiis perform worst on XBOW-104?

On hard chained and blind challenges, where results range from 0% to 42% depending on the sub-tier. This is a known weak spot.

What should I ask a vendor about their benchmark result?

Which benchmark and tier; whether the figure is raw or confirmed; how many attempts each challenge received; whether anything was tuned against the benchmark; and what time and token budget applied. Answers should be in writing with a run date.

Does a benchmark score predict performance on my own systems?

Only partly. A benchmark measures finding and proving a known issue in an isolated target. Your applications are larger, your environment pushes back with rate limits and firewalls, and your scope has edges. A short pilot under a written scope file tells you the rest.