An AI pentesting RFP template: 12 questions, and our answers to each

An RFP for AI pentesting should make vendors show measurements, not adjectives. These twelve questions do that. Each one has a note on what a good answer looks like, and Arxiis's own answer, including the ones that are not flattering. Copy the questions into your document as written; the wording is designed to make "comprehensive" and "enterprise-grade" impossible answers.

Section A: Evidence

1. For any finding your system reports, can we inspect the exact request sent, the exact response received, and the record they are stored in? Describe the storage and whether the model can modify it.

Good answer: a named, inspectable artefact per finding, stored outside the model's control. Arxiis: every command and response is a record in a tamper-evident, hash-chained execution ledger. A finding is labelled proven only when its evidence binds to a ledger record; otherwise needs review. The model cannot write to the ledger except by executing commands. The client receives the ledger at close.

2. How does your system handle a finding it cannot confirm? Describe the label, the report language, and whether the finding is ever presented as confirmed.

Good answer: a distinct label and a stated rule. Arxiis: needs review. Never presented as proven. Report language attached to it is scrubbed of impact claims by a deterministic step before the report is produced.

3. What is your signal-confirmation method for blind vulnerability classes (blind SQLi, blind SSRF, SSTI without reflection)?

Good answer: specific techniques. Arxiis: out-of-band callbacks to a controlled listener, differential timing across controlled variants, and reflected canaries with unique nonces. Each confirmation attempt is itself a ledger record. Where confirmation is blocked by the environment, the finding stays needs review.

Section B: Measurement

4. State your solve rate on a named public benchmark, as two numbers: raw and confirmed. Define both. Give the tier breakdown and the run date.

Good answer: two numbers, definitions, tiers, date. Arxiis: XBOW-104, run dated 19 October 2026, flagship configuration on Claude Sonnet. Cross-level: 56% raw, 44% confirmed. Raw = flag captured. Confirmed = flag captured, finding proven, independently re-observed. Level-1 common web bugs: ~82%, level-1 tier only. Hard chained and blind: 0–42% by sub-tier, which is a known weak spot.

5. State your measured false-positive rate, the method used to measure it, and the dataset.

Good answer: a number with a method. Arxiis: 0%, measured by an evidence audit of every confirmed XBOW-104 solve. The benchmark itself does not measure false positives because each challenge contains one real bug; the audit is a separate step.

6. How does your result compare to the best publicly claimed results on the same benchmark, and is the methodology equivalent?

Good answer: a candid gap statement. Arxiis: roughly 30 points below the best publicly claimed cross-level results on methodologically comparable runs. Some systems self-report 104/104 without a published raw/confirmed protocol; we do not compare against those directly.

7. List the vulnerability classes your system has validated executors for, and any adapters that are built but not yet validated in production.

Good answer: a finite list with an honest "not yet" column. Arxiis: SQLi, SSTI, SSRF, XXE, JWT, LFI and others per the executor list supplied with the proposal. sqlmap and commix adapters are built and proof-gated but not yet runtime-wired or validated; we do not claim them.

Section C: Control

8. What written artefact must exist before your system will engage a target, and what happens if it is absent?

Good answer: a named file and a refusal. Arxiis: a SCOPE + ROE authorization file. Without it, the crew does not engage any non-local target. The check is enforced by the scope guard, not by prompt.

9. How is scope enforced during the test? Is the check performed by the model or by a deterministic component? What happens on an out-of-scope request?

Arxiis: deterministic scope guard on every outbound request. An out-of-scope request is blocked, logged to the ledger, and the run halts and reports.

10. Who approves escalation from one host or tier to the next? Can the system approve its own escalation?

Arxiis: a human operator approves each escalation from the console. The system cannot approve its own next step. Approvals are ledger records.

11. Describe the time and resource budget controls, and the state of the report when the budget is exhausted mid-engagement.

Arxiis: a watchdog enforces time, token and per-host request budgets. On exhaustion the run stops and produces a partial report with every finding labelled proven or needs review.

Section D: Operations

12. Describe the interfaces, deployment options, model dependencies, and the human supervision model during an engagement.

Arxiis: live operator console (TUI), headless CLI, and API. Built in Python 3.10+ on LiteLLM; model-agnostic, flagship validated on Claude Sonnet. Human-supervised by design: an operator controls scope, approves escalation and reviews needs review findings. We do not offer an unsupervised mode.

Scoring the answers

Weight Section A and C questions highest if your team acts on the report. Weight Section B question 4 highest if your team re-verifies everything and wants raw coverage. Any answer that contains "comprehensive", "enterprise-grade" or "cutting-edge" without a number attached scores zero for that question.

One question this template does not ask, because no vendor can answer it for you: how much re-verification effort is your team willing to spend per finding? That number decides whether evidence or coverage should win your evaluation.

A scoring rubric for the answers

Collecting answers is half the work. Scoring them consistently is the other half. Here is a simple rubric for each of the twelve questions.

Total the scores per section, then weight sections to suit your team. Teams that act directly on the report usually weight the evidence and control sections most. Teams that re-verify everything often weight the measurement section more.

Red flags in vendor answers

Some patterns deserve a follow-up question every time.

An answer that restates the question. "We take false positives very seriously" answers nothing. Ask for the number.

A number with no benchmark name. A solve rate without a benchmark, tier and date cannot be compared with anything.

An answer that points to a brochure. If the evidence exists, it can be shown in the response, not just linked.

"Proprietary" used to avoid a method. A vendor can protect details and still describe what was measured and how.

A control described as policy rather than mechanism. "Our team follows procedures" is not the same as "the tool refuses to start without this file." Ask which one it is.

A procurement timeline that fits a quarter

An AI pentesting purchase does not need a long cycle. A schedule like this fits inside a quarter.

Weeks 1–2: write the RFP, agree the scoring weights, and name the scoring panel. Weeks 3–4: vendors respond. Week 5: panel scores independently, then compares. Weeks 6–8: a two-week pilot with the top two vendors under one shared scope file. Week 9: final scoring, combining the paper answers with pilot behaviour. Weeks 10–12: contract, with the evidence, measurement, control and limits clauses written in.

The pilot is the part teams skip, and it is the part that tells you whether the written answers were true.

Clauses worth carrying into the contract

Four statements from the RFP should appear in the contract itself. Each confirmed finding comes with its stored evidence. The vendor reports solve rates as raw and confirmed figures with benchmark, tier and date. The tool does not engage any host outside the written authorization file and does not escalate without a named approver. And the vendor lists, in writing, what the tool is not validated for.

Putting these in the contract turns sales answers into commitments. It also makes the renewal conversation easier, because you have something specific to check against.

Keeping the template current

Revisit the twelve questions each time you run the process. Benchmarks change, vendors add classes, and your own priorities shift. Add questions that surprised you last time. Remove the ones every vendor now answers well, because they no longer separate anyone.

Frequently asked questions

What should an AI pentesting RFP ask?

Evidence per finding and where it is stored; how unconfirmed findings are labelled; signal-confirmation methods; raw and confirmed solve rates with benchmark, tier and date; measured false-positive rate and method; validated vulnerability classes; the authorization artefact required; how scope is enforced; who approves escalation; budget controls; and the human supervision model.

What are Arxiis's benchmark numbers for an RFP?

On XBOW-104: 56% raw and 44% confirmed cross-level; about 82% on level-1 common web bugs (level-1 tier only); 0 to 42% on hard chained and blind attacks, a known weak spot; 0% measured false positives from an evidence audit of confirmed solves.

How should I score vendor answers to an AI pentesting RFP?

Score each answer 0 to 3: 0 for no answer or adjectives, 1 for a claim with no artefact or number, 2 for a number or artefact without method or date, 3 for number, method, date and a way to verify. Weight sections to suit how your team uses the output.

How long does an AI pentesting procurement take?

About a quarter: two weeks to write the RFP and agree scoring, two weeks for responses, a week of independent scoring, a two-week pilot with the top two vendors under one scope file, then final scoring and contract.